Move fast and break things.... or better: move fast and ship things!
Nice talk from Girish Patangay (from Facebook) on Velocity Europe 2012:
Some facts:
- Facebooks PHP "HipHopped" (compiled) binary is 1.2 GB big!
- 10,000 of servers
- Every day a new deployment
- 200-300 machines are compiling the c++ code which is generated by hiphop.
- One compile ~21 minutes (remeber the big cluster)
- Deployment with bit torrent to transfer the 1.2 GB deployment to the servers.
- Restarting the webservers is a bottleneck (~2 minutes per instance!)
- 3 steps in deployment:
Phase 1 with 6 hosts: ~15 minutes (withour compile)
Phase 2 with ~2% hosts: ~13 minutes
Phase 3 100% hosts: ~45 minutes
- InMemory configuration changes (Backends!)
- "Gatekeeper" to enable features for special user groups
