In a large pan (large enough to accommodate all pasta and veggies) over medium-high heat, warm 1 tablespoon oil until hot but not smoking. Add the shallots. Season with salt and pepper, cook stirring occasionally until shallots are slightly caramelized (6 minutes) Add garlic, cook until fragrant, about 1 minute. Reduce the heat to medium. Drain pasta reserving 3/4 cup pasta water. Add the penne, 2 tablespoons olive oil, and reserved pasta water to pan. Stir in the cherry tomatoes, dill, arugula and sherry vinegar. Cook until the arugula is just wilted, 1 to 2 minutes. Season to taste with salt, pepper and red pepper flakes.