Flying in the morning means waking up really early. It’s exhausting and inconvenient, so it deters a lot of people.

The type of people who decide to make this tradeoff, and show up, are usually particularly responsible. 

They’re punctual. They want more time at their destination, so much so that they’re willing to wake up at 4am for it. It’s probably earlier than the time they wake up for an ordinary workday.

It’s plausible there are fewer babies, which means less crying.

Since it’s so early, the airport is very uncluttered. That also means you spend less time waiting in queues.

A lot of people sleep on the plane, so it’s quieter.

In my experience, the plane isn’t very full. You might see more than one person in a row by themselves, lying down across all three seats for the duration of the flight.