Long story short, I think the following feature request would cover the core need you’ve identified here. Be sure to go and cast a vote!
On the RGB, the following feature request sounds like it would cover what you’re looking for (be sure to cast a vote here too):
As to the why on the color picker, it helps provide a consistent color palette and leverages the thousands of hours of UI/UX research Google has done with their Material Design language. That being said, I can understand the desire for custom colors. One of our goals is “simple but powerful”, so a nice approach might be to display the palette by default and have an option at the bottom to enter a specific hex value or use the full color wheel with transparency.