Hacker News new | past | comments | ask | show | jobs | submit login
Amazon built a new unit to fix its engineering culture (businessinsider.com)
112 points by firstSpeaker on Sept 21, 2022 | hide | past | favorite | 153 comments




It'd be great to read about all the things this internal DevX org has been able to improve after some time.

Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better:

* Faster turnaround time for external contributions (it's OSS)

* Ability to be relied on to get a critical fix in and tested in under an hour

* Much faster local build times

* Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much

* Everyone was happier in general

If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.


I would love to find a full-time position of just doing this kind of work. Nearly every place I've worked had some hacked together system that mostly worked, but wasn't enough of a pain for management to dedicate time to. I'd have to steal time away from the feature mill to make some little improvement, and everyone would agree that, yes, it did make things a easier for the devs, but no, we can't just have you out there working on things willy-nilly when there are features to churn out!


I created this role at my current firm. While I don't know your circumstances, here's what I did to get it funded:

* Got buy-in from my coworkers

* Collected data on how much wasted dev time there was

* Collected data on what we would gain (be specific: time to live, faster release cadence, more releases, whatever is real and appropriate)

* Collected data on what clients would gain (errors more likely to be caught before release, more features, etc)

* Present to boss, refine message, keep going up the ladder

MVP for a PoC, in my opinion:

* Come up with project goals as a team

* Sane branching strategy. Can be done at the team level if needed by making a team-level "main"

* Standalone (meaning no-deploy, "devops" framework independent) unit testing system. Use the one that comes with your language if it doesn't stink. Add unit tests to new code.

* Basic build & test system. TeamCity is easy to deploy, configure, and maintain on-prem, Jenkins supports complex use cases. Automatically run on commit to the team-level main. Allow devs to run on-demand for their feature branches. Use other systems (GitHub Actions, Azure DevOps, etc) if you have the budget.

Keep it simple. Don't worry about pull requests and approval chains at this point. Automatically build it, test it, and put it someplace it could be deployed.


We carved a role for that at my current startup (dev team ~10 people). We maintain a technical roadmap, which includes platform improvements, dev tooling, infra, etc. Other team members jump in to help out if there's something they're particularly excited about or can do well, but one person is dedicated to this work.

Most large companies have teams for this. I've found that the earlier in a company's life you add someone dedicated to this the better.


Are you hiring? ;)


A lot of companies have this role! EngProd at Google, Eng Efficiency at Roblox (let me know if you're interested), and I'm sure plenty of others as well.


I'm certainly interested! If you wouldn't mind emailing me at iam@zachgarwood.com, I'd love to learn more about this type of position.


Just sent you an email. :)


We're hiring for this role in the UK and parts of the EU: https://boards.greenhouse.io/griffin/jobs/5252742003


PE at Meta is a bit like this. When people ask what it is, my best description is "SWE that focuses on non-functional requirements" (called 'better engineering' internally)


What does PE stand for?



This kind of stuff is right my ally and I'd love to connect with someone at Amazon about how they're tackling these problems. Does anyone know someone that would like to grab coffee to chat about it?


Your note is unlikely to yield a result. It does end with a call to action but doesn’t explain why someone should respond.

It’s nice and short but needs one more sentence in the middle. This is up your alley — great! But do you want to compare notes? Do you want to get hired? What’s in it for someone from Amazon to respond (or harder, since you’re asking for an intro: why would someone send this to a friend?)

I get a lot of messages like this and usually ignore them because they lack any reason to respond.


At almost every company I’ve worked for I would estimate that if I worked on or made changes like you have described the benefit to cost ratio would be multiples larger than any of the actual features I worked on. This was always perplexing until I realized my assumption that managers were given a pool of money and were judged on the ROI they generated on that money was false.


Hmm the title made me believe this was to fix the use-and-throw-workers culture. Didn't realise it was about fixing tooling. Wonder if anything else was done to fix the disregard for wellbeing and safety?


Orthogonal concerns I think. The pessimist in me says they won't be doing much, but I think they did announce something about making working conditions better for the bulk of their workforce in warehouses, but I won't be holding my breath.


Amazon was a good gig to join (on the AWS side) 4 years ago, not so much now.

Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income...

Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out...

There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.


I'm hearing tales left and right like that; one part of the tales is that you get stocks or stock options, BUT you have to eat a shit sandwich for years before you can 'cash out'. This is the companies' golden handcuffs; they know the job is shit, but you'll get a big payout if you manage to keep it up for at least X time.

The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.


For reference, entering in July of 2016 got you $700 RSU's. Today that's the equivalent of a $35 share.


not sure what you are saying the stock lasts 4 years any new stock you getting in between will be at market price so you not gonna clear 600k after 4years but only base +new stock.. if amazon stock goes up the next 4years newcomer will get 600k too


From TFA:

> In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture.

This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot.

Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.


This type of bureaucracy is not only harmful to dev culture, it's also outrageously expensive. I've been on an engagement with a large healthcare provider and the dev culture there is very similar to Amazon, loads and loads of procedure and bureaucracy. Now i get a certain level of that at any company, but you often find yourself going: "does it really need to be this hard". Tasks that would take you an hour outside of this companies ecosystem would take you 3-4 inside the ecosystem.

It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.


While it's true that arbitrary bureaucratic inefficiencies can get out of control, sometimes "good" bureaucratic requirements can get inflated with the arbitrary ones, simply because people don't understand their reason in the first place. This leads to a Chesteron's fence scenario [1].

Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics.

[1] https://fs.blog/chestertons-fence/


What I found is almost the inverse of a Chesteron's Fence. Often times I see a complex process and start asking the reasoning behind such a process, everyone I talk to always has the same response: "you know I'm not sure why we do it this way, but it's important that we do it this way". At the end of the day you realize that no one really knows why they do a thing a certain way, and they are too afraid to change it.

I remember working for an insurance company in IT and we would always get a type of ticket that we were told to re-assign to another team. When I asked the obvious question of "why doesn't that team we send it to get it in the first place", I got all sorts of answers basically culminating in: "I don't know but it's the process and it's important". I eventually got someone to walk me through it and admit through their own answers that there was really no good reason to keep this process manual, as it must have been a holdover from when they had an older ticketing system. Even with all this it was impossible to change that process (at least while I was there) because the answer was always: "yeah your idea sounds good, I don't really see where we could break anything....but what if we do break something, let's just keep it as is".


I think you may have gotten the wrong impression about why I brought up Chesteron's Fence. It's not about blindly following a process; it's about understanding what risk the process is trying to mitigate.

“If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

This means somebody needs to first understand the reasoning behind it. If, as in your example, that rationale no longer holds, getting rid of that step is perfectly inline with the principle. I would argue that if somebody truly understood the process there would be no issue with changing it.


Chestertons fence is basically a prior: Things are the way they are for a good reason. It can be more or less applicable in different contexts, and I'm not sure I would give give bureaucratic procedures such a benefit of the doubt. On the contrary, I think that any departure from doing something in the simplest possible way has to be accompanied by strong evidence.


Here's my take. Any requirement or bureaucratic procedure is meant to mitigate some risk. You must understand what that risk is before removing that process step.

Many times that process step is just arbitrarily layered onto an existing process. If you do that enough times we get the inefficiencies that we tend to think of when we say "bureaucracy." That creates a risk in itself, namely a productivity risk. But that stems from my earlier point: when process steps are just layered on without thinking about the second-order effects, it creates a bad process by incurring outsized risk. But the converse is also true: removing process steps without understanding the rationale behind them can also increase risks.

You seem to equate bureaucratic steps with needlessness. I'm not sure I agree. I tend to think they are there for a reason, but possibly just implemented poorly. Same goes for decisions to remove steps from a process.


Heh, watch out for empire-building middle managers. "Hey how would you like to reduce your headcount by 2x" will not go over well.


Yup. This is an especially dangerous suggestion when you work for a government contractor. Since a lot of them have their profit margin capped at a specific percentage, the only way for them to increase total profit is by increasing the size of the project.

Suggesting efficiencies that would reduce headcount is a good way to get on executive leadership's threat radar.

Man, I do not miss working for a fed contractor.


It’s more like, “How would you like to get twice as much done?”

I don’t work at Amazon, but I’m sure there’s plenty for folks to do.


> Now granted many people at these places are salaried…

This shouldn’t make a difference because in theory they would be doing 2x more work which is essentially cutting their salary in half. I realize that the dev team being twice as efficient doesn’t necessarily double the unit’s revenue but it’s some multiple.


At the moment, the general impression I get from people working at Amazon is "Join for the experience, get it on your CV, move on after a year". This is even across departments and countries. The wife of a very good friend who lives in China is working there since nine months in some kind of operations role, and made up her mind months ago that she wouldn't take it any longer than a year. Here in Berlin, I just recently talked to a dev who left exactly a year after he started. He said it was interesting seeing the whole AWS machinery from the inside, and there was quite a lot to learn in terms of process and management, but shoving around legacy Java was just not worth it.


> legacy Java

I'm starting to feel very old

Is Java considered legacy now?


I just recently left a job that had a lot of pre-Java 8 still running.

That’s legacy Java.


Java came out 3 (or is it 4) recessions ago. There are entry level developers who were born after I started using it. It's a legacy technology.


For a technology to be legacy, I think it needs to be stale and falling out of use for new projects. I don't see that with Java. New releases with new language features are being released at a faster pace than ever, and companies are still building new products on it.

Java hits a sweet spot of performance, tooling, and productivity that few other ecosystems can. It's biggest competitors are probably C# and golang, and, for various reasons, it is holding its own just fine against them.


If you remember Duke, you're old. I still remember when Sun was going around holding Java events, touting the advantages of Java, with Duke shirts, mugs, etc.


I didn't mean "Java which is legacy", rather "Java code that is legacy". I think it was also running on old versions, don't remember how old though.


"legacy Java" is legacy, but it doesn't necessarily mean that "Java is legacy"


Current AWS employee of 1 year, opinions are my own

I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles.

Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.


Is it just the luck of the draw who ends up on the teams with good managers or can you dig your heels in if you end up on one of the bad teams and just refuse to stay late? Or can you easily switch teams until you find a good one?


The real issue is the manager churn. You may have a great manager for a year, then 3 more then next year who aren't great. You can work hard to burrow your way into a perfect spot, but because of churn the chances are high that your spot won't be there long and you'll start the process over. It's absolutely exhausting and ultimately one of the reasons I left AWS after 6 years.


> can you dig your heels in if you end up on one of the bad teams and just refuse to stay late

I mean, literally anybody anywhere can do this. Whether you get fired depends on your performance evaluations. Like any organization, the biggest reasons to stay late is (a) too high expectations were set, (b) failures to re-adjust to new information (e.g. we had to pull you off your project for 2 weeks but still expect you to make your deadline), (c) paging / oncall. At AWS, I'd say (c) is the biggest risk, but (a) and (b) can also be present at any team of course.

> Is it just the luck of the draw who ends up on the teams with good managers

Yeah, that's my impression. When you're already in Amazon though, you're empowered because you have a much greater ability to gather information about a team before deciding if you want to transfer. For example, message/meet with devs on the team, look at their ticket queue to see how often the oncall gets paged, and so on.

> Or can you easily switch teams until you find a good one?

I don't know about "until you find a good one", but transferring is pretty normalized. If you've been here a year or two, it's normal to want to transfer, and even before that it's not impossible. You can't transfer if you're on a performance improvement plan, but otherwise it's pretty straightforward


I know a few senior engineers / manager types who left financial services tech to work at Amazon. Most of them left within two years, and the ones that stayed quickly got out of AWS org because it was too crushing.

In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.


"In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff."

This isn't always the case. Ask a TAM. With the right blend of customers I've known some to work about 10 hours a week.


JFC, that explains a lot.


What about the hire to fire, and crazy amount of pips they do? How are they going to fix that part of the culture? These two are the biggest reason why I turn down amazon recruiters.


Churn and burn, baby!


Maybe the adversarial evaluation process has a poisonous effect?


Stripped of fluff, the main point of the article seems to be that their engineers are doing too much manual repetitive work, due to bad tooling and processes. Why would that be caused by their performance review system?


Easy: tooling and process are often seen as being of little importance because they’re not easily associated with metrics that are important in toxic review process cultures. Engineers who like working on these things are treated as second class citizens and performance reviews will reflect that.

(I don’t know that this is the case with Amazon: I’m just providing an anecdote based on my observations at places I’ve worked.)


Not just engineers, but also engineering managers and product managers. Shipping bugfixes and tooling improvements is often harder than shipping new features, and is generally less visible to the people who make hiring/firing/promotion decisions.


I’ve found tooling lags behind in places where incentives are too short term, or most of the funding comes from projects (rather than durable product teams). I’m not saying this is Amazon.


Why spend time building tools that will help others if you're competing against them?


> Why would that be caused by their performance review system?

May be because their performance review prioritizes employee contributions that impact bottom line immediately instead of building infra that might improve developer productivity?


You can't manage what you don't measure. Developer tooling quality impact on revenue is very difficult to measure. Therefore, a rationally managed organization cannot authorize any work on developer tooling.


TBF, that was scrapped in 2017. Source: worked there.

There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.


I’m not sure how you can say that given news reports like these in 2021, see https://www.theverge.com/2021/7/9/22570579/amazon-performanc...


I'm not saying there aren't other issues - there certainly are. What you highlighted is one of them. All I'm saying is that The Grand "Me-vs-You" peer review process is gone.

Anecdotally, I can say with confidence that some teams have a lot of autonomy in deciding to follow (or not follow) company-wide practices / guidance like what you've posted (obviously, only a few do it, since non-compliance isn't officially blessed). I was lucky enough to work in a team which was much better than baseline. But I try not to let that colour my memory - I heard and saw plenty of horror stories, and I saw plenty of people (including in my leadership) trying to do the right thing. Like most things, there are shades of grey here.


They're trying to fix this with dev tooling, which is the least of their problems...


Have never worked at Amazon nor do I know anyone who did, what exactly did that entail? "Snitching" on your co-workers? I'm asking because a quick web search didn't help me too much.


This is purely a PR stunt for the shareholders confidence in Andy's leadership. We've been screaming from the rooftops for years, then they hire someone even worse as HR head, Beth Galetti, who clearly shares distain for the companies talent.

Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.


"...the retail giant has pledged to become the Earth's Best Employer, not just for frontline warehouse workers but for corporate and technical employees, too..."*

* As long as it doesn't interfere with the share price and exec bonuses


Barely a day goes by without a story about how badly Amazon treats it's non-tech workers. Combine that with the stack ranking and, like you, I'm not convinced.

I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.


Are they still doing stack ranking and hire-to-fire?


Internal tooling is a mess at most companies I've worked at. At a certain scale it's worth the savings to standardize, improve, and reduce the shared friction costs.

Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.


Re: FB - still is.


I quit a few months back and hope it continues to crumble. You won’t be missed, Amazon. After 5 years and countless long nights, I was thanked with a pay cut and ignored when petitioned, even after rearchitecting the entirety of one of their floundering subsidiaries and leading a team of 10 thru it’s execution. The reason? One of my junior engineers missed a deadline. When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise.

At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.


To be frank, this compensation issue isn't a problem specific to Amazon. It's common knowledge that if you want significant raises (50%+) you have to jump among the FAANGs/Tier1 companies, mainly due to the external vs internal offer policies. An internal promo will get you into the bottom of the next pay band versus an external offer which usually puts you near the top. The compensation delta can be in the 6 figures for non-new grad roles. This holds true all the way up to director level roles in my observations.

Fair or not, this is currently the optimal career/comp progression strategy, although recent hiring downturns might change the meta(heh)game to favor internal candidates. We shall see.


very true. your manager and skip might be able to see your impact and performance, but for people who control the purse strings, the only value you can prove is a better offer.


Stories like this are the reason why I keep turning down Amazon recruiters, not the lack of shiny/speedy tools. This article is fluff. And I'm glad you got out.


Thank you for sharing your experience. I'm sure there might be some guy going "but, but, it depends on your team! I've never seen this" --but Amazon has a large variety of bitter ex-employees and horror stories.


RE: Boomerang, Amazon changed the policy recently. If you return within one year at the same level, you get the same comp you left at. A comp increase requires a new level and a full loop. The upshot is that you have to have proof of impact for that new level for boomerang to be worth it.


I'm curious, do you mean pay cut as in literally they changed your total comp or pay cut as in the share price of AMZN dropped?


>"When petitioned, my cries were ignored and I was recommended to boomerang if I wanted a pay raise."

Is petitioning a process in AWS? What does boomerang mean here?


(Disclaimer: I left Amazon about 6 months ago)

Petitioning isn't a process in AWS, but you can ask your manager for a raise if your projected total compensation isn't to your satisfaction.

Boomerang means you leave the company, get another job, and then come back to the company with much higher compensation. It's a way for otherwise happy employees to fight salary compression.


Petitioning is referred to as a "Dive and Save" internally. It is a last ditch offer to retain an employee, usually reserved for high performers who are seen as a flight risk.


A caveat there is, proactive dive and save is reserved for high performers who are seen as a flight risk Reactive dive and save is for anybody who can show a competing offer.


Slightly OT, but someone should set-up an (anonymous enough) wiki somewhere with all of this big IT company jargon, would make for interesting reading. Threads like this one, coupled with that Google comic written by an employee that was full of those jargons would make for a good start.


I’d have expected that to have started as “diving save” rather than “dive and save”.


Wow. Is boomeranging actually common at Amazon then? Why would you go back to a company that you had to leave in order to get a raise? It seems like the cultural aspects at the root of leaving would not have fundamentally changed the second time around.


Can't speak to Amazon, but while this seems surprising it's not that uncommon at big companies. One leaves assuming the grass is greener elsewhere and if/when that turns out not to be the case returning to the place you left provides a sense of confidence that you at least know the bad parts going in. And as described elsewhere it's sadly often an easier way to get a promotion/raise than grinding out the process at the company.


Sometimes it’s cause your TC takes a nose-dive after year 4 and the company has no interest in compensating for that drop. You might not want to leave but in order to keep pace with the market you’d boomerang to come back to be on par or above the new hires making 50% more than you.

I’ve also seen a scenario where someone has trouble getting promoted and leaves to boomerang back at a higher level.

Of course the company hates to give existing or recently departed employees raises of any type so they closed that loophole.


I don't know but I imagine it means to leave and then come back.


Leave and get rehired.


Boomerang is “leave and come back” (quit, get a job elsewhere, later re-apply).


>"Being an engineer at Amazon is more button-clicking than development," the slide said, quoting an unnamed Amazon developer.

You guys get to click buttons??


Lucky ducks. Development at my org largely consists of going around to every different possible "partner" to get buy in to have a meeting, to discuss approval of permissions to do something. I think the new metric used to asses an organization is PPS (Powerpoint Slides) to LOC ratio.


Is "software builders" an Amazonism, or did this guy come up with it, and either way, are they aware how weird and insular that makes Amazon sound?


Jassy has used the "builders" term for years while leading AWS. It was a central theme in many of his keynote speeches. I think it was chosen specifically because it was not an industry-standardized (and watered-down) term.

https://aws.amazon.com/builders-library/ among many other AWS docs that use the builders language.


That language predates Jassy, it was there back around 2012 at least.


Andy Jassy's been at Amazon since 1997 and headed AWS from its inception.


Predates Jassy? He has literally worked at Amazon for decades and most of that on Amazon Web Services.


Predates Jassy running Amazon.


Amazon has had an org called 'Builder Tools' or some variation on that for over a decade (maybe much longer?). That team has historically owned the systems for build, deployment, source control etc. AWS also uses the 'builder' terminology a lot in their marketing. E.g. Werner saying 'Now go build!' at the end of his keynotes, and the 'Build On AWS' campaign.


AWS console is one of the most poorly developed builder tools Ive ever encountered for a company like Amazon, from a UX perspective at least.

I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.


Sounds like a Day 2 move.


The corporate speak can be hard to stomach sometimes :)


What do people think of DevX groups in general? I'm not sold, but in my (only n=1) experience, it'd be best to try to address the cultural problems that led to the perceived need for DevX.

For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.


At first I was going to disagree but now I'm rethinking.

The need for these DevX groups comes from the fact that at Amazon's scale, there's lots of powerful tools, such as a massive internal build system and such.

So, one answer is that you need DevX groups, because it makes no sense to have each of the thousands of developers learn how to tinker/fix the huge custom amazon build system (IE empowering individual devs).

A second answer might be that there shouldn't be massive internal systems in the first place.

A lot of the brilliance of AWS was reducing the differentiation between internal and external teams as consumers of services[0]. The buildup of massive internal systems probably shows that Amazon hasn't gone far enough, and DevX failures are just a symptom.

[0] related https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-...


I can't speak for AWS ; my experience is at a much smaller company, where the DevX team was mostly busy with enforcing coding standards and other uniform "best practices" upon other teams.

OTOH if your build system needs a lot of tinkering/deep knowledge to be used effectively, maybe it needs a redesign. This would be a job for the tools team rather than DevX.


Management at Amazon was the biggest problem. The don't index for soft skills. They want procedural robots that can memorize 100 stories to cover every LP.

Culture fit matters and you can't fix the culture when thats what they are hiring for.


Brazil should be gone


Really? I miss Brazil and version sets regularly and often think about reimplementing the parts I find valuable on top of an existing package system.

Having self contained workspaces with all dependencies in any language, fine grained ability to replace any package with a local build, and the simplicity of package config (for most languages) was awesome.

Having to use _n_ language dependent build systems without interface pinning or reproducible builds since leaving sucked, imho. Checking out a package and running `bb` to get an artifact was just so satisfying.


I would check out <https://nixos.org/> if you're looking for a similar experience to Brazil.

My own opinion is that the Brazil had far too many running parts. I didn't like that reproducibility required the tool interacting with 3-4 different services, and I'm much more a fan of having the source code repository act as a single source of truth for reproducibility. Plus, using Brazil meant you had to use their entire tech stack, where GitLab/GitHub beat them by far.


Nix will get you most of this. It's got a learning curve, but I think it pays off and is not bad if you're familiar with the Brazil concepts.

Having to set up a million dependencies just to begin to start working on a codebase is a nightmare, and Brazil makes that easy.


I’ve looked into Nix at the previous company I worked for for exactly those reasons. I spent a bunch of time writing minimal dependency builders and artifact reuse for a maven monorepo where I had to tread carefully and not break the build. Nix seemed like a great way to get inter-repo dependent builds and dependency modeling.


3p packages...Python...


For people without knowledge of Brazil (the tool), this is an odd statement.

But, FWIW, I actually like Brazil.


When I taught new hires about Brazil, the common response was to revile in horror that they’d have to learn (not their favorite language specific build tool). It didn’t take long, however, for most people to see its benefits and then get past that initial bias.


It cuts both ways though. Some people are too invested in Brazil to embrace any improvements other "favorite language specific build tools" have made in the last 10? years. So it stagnates. Python support was bad, but bearable mainly due to the herculean efforts of one guy.


Thank you, I knew people hate on Bolsonaro and the like and that the elections there are coming this weekend but that shouldn't be enough reason to want an entire country to "be gone". Suffice is to say that I was confused.


> But, FWIW, I actually like Brazil.

You mean the country, not the tool, right?


Brazil (and Apollo/Pipelines, although I've heard those might be gone now?) are the things I miss the most from working there ~8 years ago.


I was confounded by Brazil for my first 3-4 years of working with it. The lightbulb clicked on just as I was ready to move on and at my next job, I missed Brazil terribly (maybe it was just Stockholm syndrome).

I eventually boomeranged back to Amazon and have found the middle ground. When used right, Brazil solves a lot of problems and day-to-day I enjoy working with it much more than not. The trouble is that almost all teams don't know how to use it and tie themselves up in knots of their own making. Third party software is also a problem. I hope Peru comes through, but so far I haven't worked with it, for good or for bad.


brazil and (legacy, non-EC2) Apollo beat any build and CI system out there


Even their interview is kind of soul crushing. Felt like I was a story teller instead of an engineer trying to get a job. I can only imagine how it is working there.


So their solution to management and process problems is to "manage harder." Yeah, this is going to go well.


Amazon built a business on hiring starry eyed engineers and then burning them to ash. I worked for them for 6 years and holy shit I'd never go back.

Their reputation is catching up with them and their lack of stock growth is highlighting their comp deficiencies.


Always amused on Blind about people posting leaving Microsoft for Amazon because TC is the only thing folks seem to focus on there. I've known dozens of Amazon engineers and I've never met one that was happy.


At the risk of sounding the "+1" bell...

I left Amazon due to it being a total shit show (for me and my mental well-being but also for the team I was on). Went to Microsoft and found it to be absolutely wonderful. I was working at 6pm one day and my manager, as he was leaving, chided me and said, "Go home to your family. Work will be here in the morning."

I have had friends/colleagues leave to go to Amazon for the TC. One even tried to get me to go over there for a PE role that likely would have meant a huge pay increase. But I value my sanity and have no plans to go back there ever.


> my manager, as he was leaving, chided me and said, "Go home to your family. Work will be here in the morning."

You had (have) a good manager. A manager who actively cares about the work life balance of their team is more valuable than a huge chunk of TC.


I've been working at Microsoft (in the Office org) for a bit over three years now and this managerial attitude is very common. I'd go so far as to say that managers making sure their reports have a good work life balance is standard.

It comes with the age of the company. Unlike start-up land, loads of people in Microsoft have families and kids, and value their family more than they value work. It's normal to leave work on time, and it's concerning if someone stays late.


How long have you been at Microsoft?


Almost 10 years.


TC seems to be the only focus on Blind. It's almost laughable, but I suspect the demographics on it skew very early stage career/young.

And yes, I've met two people who worked at AWS and liked it, essentially because they were hyper-specialized and had the rare cushy gig there. Everyone else loathed it within months, and no one lasted longer than a few years because AWS squeezes out 'productivity' with everything short of rusty pliers.


Spoke to an Amazon sr engineer a week ago.

"its incredibly high pressure and demanding, but worth it"

The money is the drive. And I worry about going for a money job that I hate because getting used to the lifestyle will mean it is that much harder to leave. I am already in that boat where leaving my current job basically means a pay cut or going to a FAANG. It is very limiting especially with family pressure.


In my experience, "incredibly high pressure and demanding" is code for:

- Hacky code with quick turn around times.

- Lots of reporting and CYA.

- Very conservative behavior (nobody's going to go out on a limb to write something new because they're too busy churning through those "high pressure" Jira tickets).

In the long run you just have an army of engineers, their backs whipped raw, toiling frantically at a mountain of tech debt.


Plenty of companies that pay well and have a good wlb not named Amazon


The Microsoft boomerang is very common. Leave, get some better pay elsewhere, then come back and get hired into the level you feel that you should have been promoted into internally (with more pay).


Amen. Worked for AWS for 6 years. 6 years I'll never get back, 6 years of abusive practices, corruption, lies, and bullshit. This is the bed they've made, let the sleep in it.


Is this just AWS that has such a broken culture or it is all other teams? Although AWS is big, but Amazon has other organisations and teams. (https://www.amazon.jobs/en/search-teamcategory).


The rest of Amazon is worse.


Agree, recently worked at a company that was acquired by them and ran the other direction.

Most devs I know see their endless recruiting emails as a trope at this point


Anybody got a non-paywalled link?


Oldest comment is the non-paywalled link; https://archive.ph/aRauw


Amazon's retail business is driven by the consumption of mass-produced garbage quality products from China that we destroy the environment producing, often abusing factory workers in the process, only to soon thereafter put them in a landfill.

They've repeatedly stolen the IP of top selling product companies, then you've got this stuff https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-c...

Not to mention all the heinous shit (unconstitutional wars, dragnet spying) they help our military and intelligence services do https://aws.amazon.com/government-education/defense/

Their abuse of their workers from warehouse to engineering is just one piece. As I tell their recruiters I'd rather eat a cactus than work for them.


Please don't take HN threads on generic flamewar tangents. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html

We have those rules because otherwise HN would basically all flamewar all the time. And not even fresh flamewars.


I don't think it's a "flamewar tangent". The article is talking about how Amazon has trouble attracting workers because of their engineering culture and my comment is saying that's just one cause of their problem. It's Amazon's behavior that's inflammatory, not my observation of it.


You changed the topic into a general kvetch about Amazon (that's a generic tangent) using heated denunciatory rhetoric (that's a flamewar tangent). Also with pre-existing talking points (clear sign of a battle comment as opposed to a curious one).

I'm not defending Amazon, btw. Their behavior can be inflammatory and your comment can also be inflammatory. That in fact is how flamewars work.


It’s general about Amazon because the problem of attracting talent is related to the behavior of the company in general, my comment didn’t change the scope of the discussion.

A discussion about their engineering culture without discussing the larger related problems is myopic. It would be like discussing problems with the logistics of moving people to the gulags and you’re saying we shouldn’t get off the topic of transportation.

There shouldn’t be any flame “war” because everything I said is correct. I don’t think there’s any popular defense of these behaviors from Amazon, flame wars are characterized by popular opposing schools of thought.


Sorry, but I think that's a stretch. Your comment changed the scope from "specific new unit at Amazon" (the article's topic) way past even "engineering culture" to just plain "boo Amazon" - a heap of negativities with only the name in common. That is a classic generic tangent in the sense that I'm using the phrase.

Moreover, you listed them using the denunciatory political rhetoric that is most antithetical to the curious conversation we want here. No doubt BigCo deserves it, or at least you feel they do (and I'm sure you have many good points), but we're trying to optimize for one specific thing here (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...).

It sounds like you define flamewar more narrowly, which is fine—I'm talking about the kind of high-indignation-low-information internet discussion that evokes angry-reflexive responses from people instead of curious-reflective ones. Internet threads have a strong tendency to end up in that state, and it's exactly the opposite of what we're going for here (see that previous link), so we end up expending a lot of energy to try to stave it off.

People often mention "correctness" (or "truth" or "facts") to justify breaking the site guidelines, but of course there are plenty of ways to be aggressive, flamebaity, or just plain off topic* while nevertheless saying correct things. Correctness is great, but those other qualities are undesirable regardless of how correct you are or feel you are. Here are a couple recent explanations on this last point if you or anyone want more:

https://news.ycombinator.com/item?id=32909407

https://news.ycombinator.com/item?id=32697044

Lots more here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

* Not all offtopicness is bad; whimsical offtopicness is usually fine. It's the generic tangents that are bad, because they drag down discussion quality.


Unless they turn around soon, I don't see how their goose isn't cooked in the long run. The site has become too filled with garbage products, hijacked listings and fake reviews. Once the people who think they've never received a counterfeit realize there's no way they can really know and start to question why the products they get from Amazon break earlier than usual or have questionable packaging, it will be hard to undo the loss of trust.


As long as they pay top dollar for engineers, I think some folks will happily (or unhappily) work there. AWS is a behemoth and it has its tentacles in almost every piece of internet based software we use today, it’s not going anywhere unfortunately.


> As I tell their recruiters I'd rather eat a cactus than work for them.

That's not much of a statement https://en.wikipedia.org/wiki/Nopal


Cactus tacos are very tasty!


>As I tell their recruiters I'd rather eat a cactus than work for them.

lol. whats their response?


> As I tell their recruiters I'd rather eat a cactus than work for them.

wow, hardcore.


there's junk on every retail platform, but i disagree that amazon retail is all garbage. myself and peers have noticed an increase in quality of chinese import over the past 5 years. to the point that i would seriously questions buying a brand name domestic product over the chinese/asian import off amazon. i think the quality gap has been slowly closing.


I’ve stop buying name brand on Amazon (Nike, Apple products, Hanes etc) a few years ago due to poor quality assurance and potential for counterfeit — cue joke about how Nike and counterfeit Nike are made in the same sweatshop. I would happily buy counterfeit Nike but only if it’s heavily discounted. I don’t want to be tricked into buying second hand clothes, useless junk or counterfeit products. However you bring up a good point, some of products on Amazon from Asia are just fine if you’re looking for basics like T shirts and Athletic wear. But outside of that would highly suggest people buy electronics and brand name clothing elsewhere.


i guess your comment really brings up what you should be buying on amazon. tcl roku tv for example, had one for 6 years no issues until i broke it moving. household goods ive always had good luck, as well as some basic decorations/furniture. whole foods delivery, yes please. i simply dont understand where the distaste comes from besides political feelings towards the company.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: