Arduino electronics can be an expensive hobby, but it doesn't have to be. If you want legitimate parts shipped in a timely manner you can buy them from Sparkfun, Adafruit, or some other respected outlet.
However, if you don't mind waiting about a month and only being 90% sure that parts will work, you can buy them directly from China on Ebay. I've bought most of my components on Ebay over the last five years, and I've only had a few items that didn't work.
Below I've listed what I search on Ebay for each item.
Note: The hyperlinks might die at any time. Ebay links aren't reliable.
Arduino Uno - look for one around $4. I've bought this one a few times from lucky.warm.
Arduino Mega - search for "ATMEGA2560-16AU". This will show you the off-brand megas, you should be able to find one for about $8. I've bought this one.
DC Barrel Jack connectors - buy twice as many as you think you'll need, these are super useful to have. Look for the ones with green terminals. They should be around $2 for 10 of them.
Neopixels or individually addressable LED strips - search for "WS2812B led strip" if you want the 5V strips, "WS2811 led strip" if you want the 12V strips. Note that WS2811 are only addressable to every 3 pixels, not every 1 pixel. Should be around $28 for 5M 300 LED's non-waterproof.
Power supplies - look up the volts and amps, with amps rounded to the nearest 1 (or 5 above 10A). So "5V 10A power supply"or "12V 4A power supply". Be sure you test these before hooking them up to your project, they don't always put out the rated voltage.
Resistors - unless you need high wattage resistors, just buy a set. You can get a 2600 piece set with 130 different values for $12. Search for "resistor kit"
Capacitors - when starting out, buy a capacitor kit. You can get a set of 125 for $10 ish. Make sure you get capacitors rated above the voltage you're using, I've blown a capacitor and it's really scary.
Sensors - look up "[the sensor] module" or "[the sensor] Arduino". Adding module or Arduino to your search helps it find the component part instead of a nice assembled solution.