Okay, now what does Kubernetes have to do with ORMs? Why does it even have a say in what applications we're running on top of it? Am I missing something?
In another tangent, I still use raw SQL queries in most cases over ORM. That's just me. May be I'm a control freak or I just dont know how to magically use the abstraction of ORM and still get the most optimised results.
In a complex environment, that will not always work.
We use an ORM at work for all simple CRUD stuff, but when you want something more complex and performant, we end up writing sql anyway.
I am not. I am sorry if that sounded ambiguous. By "By hand" I meant SSH into a container and then pasting raw query into SQL client. ORM will not always help you achieve what you want so writing queries is alright, but you need to have tests too. You can create a command for your app (like a Django command for example) that performs the query, but you also create tests that prove it is doing what you think it is doing. Then you run this command in the cluster. This way you can replicate it to different environments etc.
Apparently raising a good, quality child/person isn't a respectable ambition and should instead be avoided in pursuit of the rat race know as a career... gotta love this mass damnation of mother/fatherhood!
Who are you arguing with? You don't like poverty? Neither does anyone else.
And if the metrics don't appeal to your preconceived notion of how the metrics should work, you want to change the metrics? You want metrics to not notice changes in demographics?
What's fair about perpetuating the systematic discrimination that women in tech face?
And in what way is this special treatment?
Companies use a variety of different metrics by which to determine which schools they should recruit from. "We want to increase our diversity" is a reasonable metric to use, and "recruit from colleges with an above-average percentage of people in a minority group we've chosen to view as important" is a reasonable way to accomplish that.
I shouldn't have to point this out, but diversity is more than just a buzzword. Increasing diversity on the development side improves the final product, for a variety of reasons.
I wrote a collection of Dockerfiles for images running Python 2.7 or Python 3.6 + Selenium with either Chrome or Firefox and using Xvfb for the X display (necessary for running Selenium headlessly).
In short - this gives you the ability to pass the output of one lambda function to the input of another lambda function.
An example of one that I've written to regularly create a new copy of our Production RDS database in Ireland as a Staging RDS database in Oregon.
1. Cloudwatch Event starts Step Function on the 15th
2. Copy last Production snapshot from Ireland to Oregon
3. Restore this snapshot as a new RDS instance (It will fail until the snapshot is available and retry with exponential backoff - this is a step function feature)
4. In parallel:
- Add tags to the instance (Once it's available)
- Delete the snapshot copy (When finished restoring)
- Modify the new instance with security groups and subnets
- In parallel:
- Run a SQL query to anonymize all of PII columns for GDPR compliance as data has now left the EU.
- Call out to the Cloudflare API to update our DNS entry with the new RDS endpoint.
- Delete the old Staging database instance
Do you run into problems with Lambda's 5-minute maximum execution time for those kinds of operations? I'd like to do something similar to this for both RDS and DynamoDB, but the execution time will often surpass 5 minutes, meaning I'd have to run a Step Functions worker on EC2 or ECS. That opens up a whole bunch of complexity with managing the worker code and its deployment, which I'd rather avoid if possible.
With the current implementation; no problems hitting the limit. As mentioned in my below comment, our query for anonymization would be the heaviest - but it's designed to be quick as we don't care about unique values for most data.
If we did though - Fargate is a great solution for it, but you wouldn't be able to feed data back into the next step without some additional complexity - Maybe have the next step pull an SQS queue, or an S3 file, or look for a database entry, etc. as it's next bit of data that it needs - and just fail until it finds it, and once the Fargate (Or whatever) has done it's job and placed it in your method of choice, then it could continue.
> Run a SQL query to anonymize all of PII columns for GDPR compliance as data has now left the EU.
Do you ensure the values you replace with 'make sense' in the context of the application? i.e are names turned into fake names?
If so, I would love to hear more about you handle the complexities of this. If not, it's still a wonderful pipeline that I'm putting my ideas box, thanks for sharing.
Nope! I probably could with the Faker library; but we don't care about that - and to do so would be a much heavier query. My query looks like this, so it runs extremely fast and isn't an issue on Lambda:
`UPDATE Users set FirstName = 'FAKEFIRSTNAME', LastName = 'FAKELASTNAME', StreetAddress = '123 FAKE ST.', Zip = '10001', PrimaryEmail = Cast(NewId() as varchar(36)) + '@x.com', Phone = '555-555-5555')`
That is a old armchair argument made from those in an ivory tower.
If it was YOUR business, you would want to control your dream, your vision -- not a bunch of rich folks who could decide to kick you out from your dream if your main goal is not to _remain profitable to the shareholders_.
