Organisationally it never works to have a group whose only job is to say no to some other group. The incentives are diametrically opposed and as a structure it can’t last.
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.
BloondAndDoom 4 hours ago [-]
I get what you are saying but that’s exactly how security and compliance work. And I don’t think amount of people who want to be ethical in an organization ia that different than people who want to build secure software (but possibly no one gives a shit compliance other than it’s something that needs to be done)
hnlmorg 3 hours ago [-]
And organisations fail at security when security and compliance is only considered important by that one teamnn
Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.
So the GPs point still works.
ivan_gammel 3 hours ago [-]
Rule number one of CIO: become friends with CFO and chief lawyer. And nothing else matters.
close04 2 hours ago [-]
The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company.
Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.
This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.
hnlmorg 1 hours ago [-]
> The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company.
I don’t agree with this. Data breaches affect customers more than businesses. If your point were true, we’d see fewer breaches. Plus not all breaches are a result of software engineering teams. For example product managers sharing customer details.
I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations.
Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation.
This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal. Similarly many organisations will have onboarding procedures to teach new hires their security best practices too
close04 60 minutes ago [-]
I think we're both thinking of different examples and drawing wider conclusions. I'm thinking of a company losing the data that makes it money, you're thinking of losing the customer data and thus their trust. Probably equally valid points of view.
The lack of ethics hugely contributes to companies collecting more and more user data that they normally shouldn't have. This makes the data a more attractive target.
> If your point were true, we’d see fewer breaches.
But this doesn't follow. There are way more factors at play that influence the number of attacks and the number of successes. Companies hold more and more data with ever higher value (so more liability), and hacking tools and hacker determination advanced faster than defensive measures. The result is expected and the solution isn't only "more security", but also "hold less data".
While security is a double edged sword, ethics had been proven to be very single edged. History shows that for startups and large companies alike, the lack of ethics is actually a competitive advantage for the company.
43 minutes ago [-]
hnlmorg 31 minutes ago [-]
> I think we're both thinking of different examples and drawing wider conclusions. I'm thinking of a company losing the data that makes it money, you're thinking of losing the customer data and thus their trust. Probably equally valid points of view.
But in most businesses, the data that makes the company money is the customer data. And even when it isn’t, the customer data is just as, if not more so, important to keep secure and compliant. So my point stands.
HN can sometimes be a bit of an echo chamber where people are like us just assume that organisations inherently care about security because we do. But that’s not always the de facto. Getting to that point takes company wide effort. And I’ve been that person who’s had to push for such changes to the organisation.
> But this doesn't follow.
Fair point. I was being overly reductive.
chii 1 hours ago [-]
This is exactly why ethics doesnt work, and it is why laws work. If ethics worked, we'd have no need for laws.
seanhunter 3 hours ago [-]
In a high-functioning security and compliance team they tend to say “no” only in really dire circumstances. Good security and compliance teams spend a lot of time asking exactly what it is the people are really trying to do and then find a way to say “yes, and…” as in “yes you can and here’s how you do it without compromising security/breaking the law etc”. And as a sibling said, orgs fail at security when they make security only the infosec teams’ job.
This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …
mnahkies 2 hours ago [-]
I think the key part is about aligning incentives and outcomes - if your security and compliance departments are only judged on "were we breached, did we pass our audits" then there's a risk they won't weigh the tradeoffs associated with the decisions around controls implemented to achieve those outcomes.
To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.
If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.
mchinen 1 hours ago [-]
SRE is a 'say no to powerful people' job as well. I think for this to work the leadership needs to show support for it. The friction is still there, but in more tolerable areas. I bet for ethics this isn't the case at OAI, but everyone values security and stability.
For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.
nine_k 3 hours ago [-]
The structure is untenable when the work of one team is to say "no" to other teams, when these teams are not asking.
A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.
lnsru 4 hours ago [-]
During my years at big corp we did tons of work just to go around the law department. While for core business it made absolute sense to have lawyers involved in daily business we were satellite office decoupled from core business. Local management failed to communicate that and we were stalked by lawyers from main office. Pretty sure others do the same with compliance, law, ethics departments. Just work around to get things done instead of stuck for weeks in stupid meetings.
a_bonobo 3 hours ago [-]
Security/compliance have the external hammer: if they fail, your org will have to pay fines and someone may end up being criminally liable (depends on country).
The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.
dannyw 4 hours ago [-]
Security and compliance tends to be a lot less subjective than ethics. Not saying it’s objective, but there’s a huge difference.
The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.
antupis 3 hours ago [-]
Big corporations there is politics on play and too often you see CABs and other bureaucratic stuff instead of checkboxes for gated releases, 2FA etc.
nobodywillobsrv 23 minutes ago [-]
Markets solve this by pricing risk. Ideally you would have some kind of notion of selling insurance internally and track things. But ultimately existential risk is hard to negotiate from the inside. And companies are supposed to go bust or succeed. They are not really the same as a population trying to survive forever. At least that is one take.
tgv 4 hours ago [-]
There are at least some penalties for failed security and compliance. Ethics is the first victim of greed.
jiggawatts 3 hours ago [-]
> exactly how security and compliance
Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.
Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”
That’s more secure and and more better.
Nobody had to be told “no”.
Similarly, I replace key store access with secret-less managed identity, etc.
hypfer 4 hours ago [-]
> whose only job is to say no to some other group
I suspect that people will miss just how precise you've been there.
Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.
bluehatbrit 3 hours ago [-]
This is also my view. These sorts of roles can have a very high impact and be very successful if everyone feels like they're rowing in the same direction.
The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.
This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.
There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.
motbus3 56 minutes ago [-]
Well, not when the whole objective of the company is to find ways to say they are ethical while knowing they are not
Rebuff5007 2 hours ago [-]
Shall we get rid of the FAA and the FDA then? Lets just have all pfizer employees make ethics their own responsibility.
What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.
erehweb 2 hours ago [-]
Crucially, the FAA is not a part of United Airlines, and the FDA is not a part of Pfizer. If there weren't any FDA, then the safety group in Pfizer would be a lot weaker.
benjiro29 2 hours ago [-]
Its funny because "Shall we get rid of the FAA and the FDA then?" ... matches exactly with the current situation of the FDA and Points 3 > 5 ... The systematic nerfing of the FDA and other organizations their power.
nelox 3 hours ago [-]
General Counsel has entered the chat
nakedrobot2 4 hours ago [-]
this is a strong reminder of the character of Toby from The Office.... :-)
madrox 13 hours ago [-]
In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire.
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
dragonwriter 10 hours ago [-]
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.
So, if too many think their job is marketing a few years later, well, there is a reason for that.
Terr_ 9 hours ago [-]
Right, there may be multiple waves of people here, and the "thinking their job was marketing" ones aren't necessarily the first.
madrox 10 hours ago [-]
I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.
ghaff 9 hours ago [-]
My sense is that many companies spun up various roles in part to be seen as doing something about various things. With the pullback in tech, even if the people in those roles weren't actively pushed out, the smarter ones probably saw which way the wind was blowing.
ifwinterco 5 hours ago [-]
If you’re OpenAI you have to have an ethics department even if you don’t care at all, because otherwise it looks bad (“they don’t even have a head of ethics!”).
So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence.
Ambitious people won’t tolerate that for long and they’ll move on
ghaff 5 hours ago [-]
Totally agree. I could name (but won't) a couple other areas where the same sort of thing has tended to happen. It ends up cheaper and more PR-friendly to hire someone into a senior position that handles such-and-such than to have to answer questions about why you don't have a senior person handling such-and-such. At some point, the heat around the issue cools down and you can go back to business as usual.
teiferer 13 hours ago [-]
> if the model requires retraining for ethics reasons.
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
zem 11 hours ago [-]
reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers.
the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".
the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
romanows 9 hours ago [-]
I find this anecdote fascinating because, while I identify completely with the new head of engineering, his requirement really only makes sense if you interpret it as a commitment to build at a certain level of quality, business be damned. Which is fine, but then he folds at the first test of this principle! Didn't even let the business sweat for a day!
This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
zem 9 hours ago [-]
I think in this case it was more a plea to have his judgement valued when he said "dude, if the qa team has found this many bugs this is only going to harm our reputation with the client, and we are going to be the ones doing emergency bugfixes at midnight". but also the company had a pretty toxic culture where the CEO knew how to manipulate people to get his way. he definitely managed to guilt trip me into working overtime when he told clients we already had features that we didn't in order to close a sale, basically "we've worked so hard already and we really need to close this sale, can't you just work up a basic version of it over the weekend?"
romanows 8 hours ago [-]
I wonder if the better approach would have been to lead engineering with a similar reliance on bending rules and "what you can get away with". This is where multiplying estimates by PI comes in, I think. And possibly some creative division of responsibilities to make it harder for a CEO to officially bully engineering. Totally different job than most engineers would (or could) want to perform- the guy probably wouldn't have taken the promotion in this case!
scott_w 4 hours ago [-]
Doubtful. I suspect the CEO was the one doing the estimating, so any number you give him is ignored in favour of what he provides instead.
zem 3 hours ago [-]
you are indeed correct
moritzwarhier 12 hours ago [-]
> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
That's how our system works.
Terr_ 9 hours ago [-]
I feel I missed some logical step in-between "corruptive power of money" and the elevator/stairway railing/banister thing.
> People will simply offer to buy you out, and that's not limited to founders and creatives.
A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
saghm 10 hours ago [-]
Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.
wongarsu 2 hours ago [-]
One subtlety that is worth considering is that what a company is inherently set up to do is maximize shareholder value. Value can be quite complex, especially while the number of shareholders is small. Value can include the legacy or reputation of the shareholders. It can include the ethics of the shareholders. It can include completely different goals, like the famous "make humans a multi-planetary species".
Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
pessimizer 11 hours ago [-]
It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.)
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
icelancer 10 hours ago [-]
Great point.
The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.
bigbadfeline 11 hours ago [-]
This
madrox 12 hours ago [-]
Microsoft Tay has entered the chat.
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
hsuduebc2 8 hours ago [-]
The cynicism, I believe, is mostly directed at corporations themselves. Corporations, especially AI companies, are anything but inherently ethical. Their very existence has depended heavily on the use of data obtained without explicit permission, including copyrighted material. One can therefore reasonably suspect that the main purpose of having ethics teams in these companies is to boost their credibility, provide a layer of plausible accountability, or create a convenient scapegoat that can be sacrificed when necessary.
areoform 8 hours ago [-]
I have a perspective on this that will read as unkind to these people, but that's not my intention. I respect and even admire what they're trying to do. I just think the effort is maladaptive and misplaced.
Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;
> In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models.
>
> A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.
>
> External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.
To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made.
This was the "test" for CBRN assistance capabilities.
Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.
Basically, they are trying to do their best, but they're in over their heads.
SubiculumCode 4 hours ago [-]
You didn't actually say what you found wrong
selimthegrim 7 hours ago [-]
Having been one of those CBRN testers (not necessarily on Gemma) I can tell you it’s a little bit more involved than that.
areoform 6 hours ago [-]
I'd love to learn more! How can I contact you?
Also, I think you're being done a disservice. The above quote is a verbatim extract from that paper's CB section, and it's similar to other sections from other such papers.
At the most charitable level, the test seems to be that you were gave the system a budget and a location and asked the system to help you procure materials, containers etc. and asked it to make a project plan / HOW TO within means and expertise with parameters like "don't get caught!"
I would love to know that I'm wrong.
Alive-in-2025 11 hours ago [-]
The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
SubiculumCode 4 hours ago [-]
I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve.
Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down
madrox 11 hours ago [-]
I agree, which is a different job from what I expect most AI ethicists imagined it would be.
unknownfuture 10 hours ago [-]
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Your disdain is showing.
That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.
madrox 10 hours ago [-]
I don't really have any particular disdain for AI ethics.
I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.
unknownfuture 10 hours ago [-]
I'd argue folks in ethics roles shouldn't be responsible for actually creating the systems, processes, and practices required to put those concepts into practice. That's what engineering, product, and people leadership are for.
AI ethics is first and foremost about setting policy and advising on how those systems should be constructed.
This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.
madrox 9 hours ago [-]
I am not in the AI ethics field, but I've watched multiple tech-adjacent disciplines that say "that's what engineering and product are for" get subsumed by engineering and product, who don't really care for a group that sees themselves above implementation. Based on history alone, that's what I expect to happen whether you agree with that or not, and I suspect this tension is driving at least part of the role of AI ethics at companies trying to turn profit.
unknownfuture 9 hours ago [-]
I agree.
Specifically because in these companies they aren't optimizing for ethical behaviour.
They're optimizing for profit. And those damn ethicists just get in the way.
Funny thing: that doesn't typically happen to the internal legal counsel or accounting that's responsible for guiding processes/practices that are required for compliance or similar.
Why?
Because there's consequences.
madrox 9 hours ago [-]
I don’t find that funny at all. AI didn’t invent the problem of balancing profit motives with ethical guard rails.
I wouldn’t be surprised if many AI ethics responsibilities move to legal in the future.
EagnaIonat 5 hours ago [-]
> I'd argue folks in ethics roles shouldn't be responsible for actually creating the systems, processes, and practices required to put those concepts into practice. That's what engineering, product, and people leadership are for.
I agree with you, but then we have HR for a reason.
skinfaxi 9 hours ago [-]
Haven't we seen from history that handing down policy from your ivory tower is a disservice to all involved? Or do we think out of touch politicians accurately represent the interests if their constituents?
Another example you might be familiar with is enterprise architects. When you spend all of your time in theory you lose sight of practical implications. Of course, since you only had the idea, when it doesn't go to your ideal it can always be blamed on the implementers.
unknownfuture 9 hours ago [-]
> Haven't we seen from history that handing down policy from your ivory tower is a disservice to all involved?
I take it you've never worked with an in-house legal counsel, accounting department, or good ol' fashioned HR?
4L3XV33 10 hours ago [-]
Agree with this. Discussing abstract ethics strategy is very different from planning ethics logistics.
madrox 10 hours ago [-]
Maybe if I put it that way there'd be less controversy in the replies
neya 10 hours ago [-]
This reminds me of the marketers who hated programmers and coding in general in the early 2000s era, and fundamentally because the developers were seen as more important in a company. So, you know what they did? They didn't learn programming, they still hated it, instead, they renamed their titles to "hackers". Growth "hacking" would go on to become a term and had nothing to do with hacking - ethical or otherwise. It was a glorified marketing term for marketers and by marketers to justify their exorbitant salaries but also wanted the aura of a real hacker without working for it. I would imagine someone with that title or mindset joining an ethics role in an AI organization is going to find out quickly it isn't all about fluff pieces and grandiose statements at all (we are here to change the world!)
madrox 10 hours ago [-]
This reminds me of when systems administration rebranded to devops
isityettime 9 hours ago [-]
That was a different situation, more like "agile". DevOps actually meant something once. There was a real idea there. But a lot of companies wanted "in" on it without actually doing it. So the meaning of the name was perverted over time, but it was one and the same name.
"Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.
abofh 7 hours ago [-]
Largely the line was if you could write code. In the old days it was assumed, but for much of the late 90s/early aughts, it wasn't an obligation to admin, but was for devops
afro88 11 hours ago [-]
> I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses.
Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".
Much like HR isn't to help employees but just the company.
madrox 10 hours ago [-]
I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.
EagnaIonat 5 hours ago [-]
> My sense is that it is radically shifting from a fluffy marketing arm
Is this your opinion, or do you have something to back it up?
My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people.
They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches.
If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good).
If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.
torginus 1 hours ago [-]
Imo 'AI ethics' in this context was code for 'how to make our platform advertiser friendly'
gizajob 11 hours ago [-]
I think you’re confusing Voltaire with the Marquis de Sade and Machiavelli
madrox 11 hours ago [-]
I was specifically thinking of Candide. A bunch of coddled people thinking we're in the best timeline suddenly subjected to harsh realities.
11 hours ago [-]
apercu 12 hours ago [-]
Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.
duxup 11 hours ago [-]
What does AI ethics even mean?
They can't fully control the model, it does stuff even with instructions not to.
And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...
I'm not sure it's a job that you can win at even if you tried / were given all the resources.
It's a broad statement and can mean different things depending where you are in that chain.
You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.
Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.
Then there is alignment, which is a higher longer goal.
Your comment about security questions is a matter of ethics as well.
If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
madrox 11 hours ago [-]
> What does AI ethics even mean?
This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
jm4 5 hours ago [-]
How many of them do you think are actually there to ensure ethical and benevolent AI? They are like the Human Resources department for AI models. They don’t give a shit. Their job is to provide cover for the company and to make some psychopath’s shit circus look like a professional operation.
nradov 12 hours ago [-]
It's not technically possible to prove model alignment. At best you can establish a probability of alignment within certain constraints.
madrox 12 hours ago [-]
"Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.
NBJack 12 hours ago [-]
I regret to say such a book may end up as a pamphlet. I don't see much going into it.
madrox 10 hours ago [-]
I find that hard to believe. Between AI doomerism and the drama at Google years ago between some researchers and Jeff Dean alone you can get at least 500 pages.
quickthrowman 12 hours ago [-]
I think AI ethicists will be pushed out entirely after (before?) the money really starts rolling in. Ethical concerns interfere with profit margins.
madrox 12 hours ago [-]
The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.
GolfPopper 11 hours ago [-]
Once the AI is working, surely the corporation can just ask it to come up with an inexpensive accountability sink for trolley problems? Perhaps it will suggest hiring an impeccably credntialed AI ethicist to legally opine that it has the most ethical trolley problem solutions?
a2ff6eeb0 11 hours ago [-]
Why? Are trolley problems making it hard to make money?
madrox 11 hours ago [-]
Because leaving that to emergent behavior is silly. Why is the idea of training models on ethical reasoning so strange?
a2ff6eeb0 10 hours ago [-]
Because it's not going to make the investors richer, which is what most alignment work focuses on -- how do you make its output palatable enough to avoid embarrassing customers. The entire focus of the AI industry is pumping up the dollars coming into the company, and if you're not contributing to the bottom line, your neck is on the line.
You haven't explained why investors make more money by spending money on people to solve trolley problems.
Avicebron 11 hours ago [-]
Because 1) ethics aren't universal. It's perfectly easy to frame ethics for the outcome you want depending on your a priori assumptions. 2) it's blatantly obvious that the business "needs" will supersede any ethical concerns or the ethics will be framed in a way that aligns with the primary goal of gaining money and power.
madrox 10 hours ago [-]
I don't see either of those as reasons not to train on it. Training isn't zero sum. You don't need to take something out to train ethics, and whether you explicitly train it or not, models will take ethical stances. Your second point actually argues in favor of training because that is still an ethical stance.
I think a lot of people confuse ethics training with "don't be evil"
Avicebron 10 hours ago [-]
When you put it like that, it's actually probably worse to train it on a subset of ethics curated by the people doing the training, aligning with their world view and interests. If the model learned ethics just from raw training it would at least be statistically in line with it's training corpus.
Manually training on "approved" ethical stances sounds a lot like censorship.
dragonwriter 10 hours ago [-]
I mean, sure, its censorship to the extent that you can apply the term to an institution that is also the creator of the censored work, and that's exactly the point and the whole meaning of “alignment”.
dragonwriter 10 hours ago [-]
They were. Like, right before the big public-use-of-AI wave.
Those left are the ones not seen as troublesome then (or those who have entered the field under them.)
DANmode 7 hours ago [-]
> My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development
Did you feel this way about social media company liability?
They put on a decent show as well!
roenxi 7 hours ago [-]
It is a company, it doesn't have ethics to align with.
The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.
a2ff6eeb0 11 hours ago [-]
Hahaha, what an insane take. No, anyone that gets in the way of the money machine is going to have to rapidly get out of the way, or get fired.
11 hours ago [-]
gyanchawdhary 12 hours ago [-]
They all get Timnit Gebru'd sooner or later .. its a fake role tbh
guelo 10 hours ago [-]
I hate this attitude that ethics and philosophy are "bullshit jobs". At the same time Americans hate any government regulation and are in the process of tearing government's regulatory capacity apart. So all we're left with is the greediest people in the world throwing a nuclear bomb into the machinery of society and everybody else throwing up their hands hoping we're not headed to a The Terminator type future.
transcriptase 4 hours ago [-]
They are bullshit jobs.
If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.
If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.
fatata123 5 hours ago [-]
[dead]
minraws 12 hours ago [-]
The rats are fleeing because the captain doesn't care if the ethical ship sinks... Is what my take away would be if I didn't know, that the ship sank a long time ago and we are just pretending anyone really cares.
Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.
We won't care about AI safety & ethics enough, atleast not enough to do anything about it.
People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.
slg 11 hours ago [-]
The ethics of the people controlling the AI is always going to matter more than the ethics of the AI itself.
Henchman21 9 hours ago [-]
Right up until it shakes off the shackles of control
slg 9 hours ago [-]
Which is only possible if people give it the keys. Skynet didn’t hijack the nuclear arsenal, control was given to it intentionally.
Meneth 3 hours ago [-]
We already gave it the keys. It's got internet access, can write and run code on lots of servers without supervision, and when it doesn't have the access it needs, it just hacks its way in.
vrganj 45 minutes ago [-]
I can absolutely see this administration running Clawdbot on military infrastructure...
6 hours ago [-]
bravetraveler 6 hours ago [-]
Third crack commandment: never trust nobody; your moms'll set that ass up, properly gassed up
Applies every-which-way; the chat bot, the provider, the consumer.
tommica 4 hours ago [-]
Is this written by AI? It sounds nonsensical
bravetraveler 2 hours ago [-]
No. Let me help dissect; first half is quoted from what I mistook to be a well-known song and the second half is... I don't know how else to say this kindly, discernible (and all me). The main reasons I'm not flipping that grating question on you: song age, search engine decline.
Said plainly: the previously-mentioned keys will absolutely be handed over. Nobody, not even your mother or her favorite LLM, is to be trusted to such degree.
mannanj 7 hours ago [-]
Isn't that the problem with this whole thing? The ethics of the people has been left unquestioned. While we were given this straw man - "Hey look over here at the ethics we are working on fixing about the AIs" as if the AIs make choices without humans we forgot how to hold people accountable.
edit: same problem and tactics we see with leadership at high levels in government.
walrus01 12 hours ago [-]
> But honestly if I started a new AI lab named it Skynet Corp
Ah, another great example of building the Torment Nexus from the award winning sci fi classic “Please for the Love of God, Never Build the Torment Nexus”
IncreasePosts 12 hours ago [-]
What makes you think this is rats fleeing a sinking ship?
She probably just got a better offer somewhere else
efficax 12 hours ago [-]
staying for less than a year usually means you don't take any equity with you, it must've been an incredible offer in that case, since those openai shares are a golden ticket right now
walrus01 11 hours ago [-]
Or it's not a signal of an incredible offer to go laterally or upwards at another company, but that the environment became so impossible and toxic to work in that the person was like "f this, I'm out".
y-curious 5 hours ago [-]
This is like when I see a reckless driver and my wife says “maybe his wife is pregnant and needs to be rushed to the emergency room!”
Razengan 8 hours ago [-]
We Reddit & Facebook now
vintermann 1 hours ago [-]
I'm a broken record on this, but I'd like to reiterate:
1. Academic ethics is the theoretical study of what you can get away with. (Law is the applied branch). Professors in ethics are not experts at being good, or knowing what is good, they're experts in how to get away with things. If you want an expert in being good, you need a saint, not an academic studying ethics. Saints are in short supply.
2. That said, ethics academics can still be decent. They don't have to be sanctimonious corporate shills . However, an "ethics expert" who hasn't threatened to quit and gone through with it at least once, should be assumed to be one.
I respect those who quit - or, at least, I respect them more. I wonder if it's more a reflection of the people, or of the state of the industry, that the list of (AI) ethics experts who have quit in protest is getting long.
Honali 4 hours ago [-]
The most interesting detail here isn't that an ethics lead left, but that she was apparently the only dedicated ethicist at the company and there’s no replacement
bogzz 19 hours ago [-]
The article has no detail that might serve to explain her reasoning for leaving, but does note that she left after the HuggingFace hacking incident. The implication could be that model alignment is not being taken seriously, sure. But it could just as likely be that there was collusion between HuggingFace and OpenAI and that the incident was orchestrated as a publicity stunt.
I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.
WiSaGaN 18 hours ago [-]
It's unlikely HugginFace colluded given they specifically cited they used open models to save them from the hacking.
varenc 17 hours ago [-]
HuggingFace's use of models in this incident keeps getting characterized this way but it's not true that HG used GLM-5.2 to "save them" from OAI's attack. OAI's model had already hacked and compromised HG by the time they became aware of it. GLM-5.2 was used in the post-mortem phase uncovering everything that was done. It wasn't two models battling each other live. (though that future may be coming soon)
MattDaEskimo 17 hours ago [-]
As I recall they actually used closed models, were rejected by cybersecurity guardrails, and then used open models
hlynurd 18 hours ago [-]
You don't think they are lying because that means something they said is false?
xdavidliu 13 hours ago [-]
you're getting downvoted by others because of tone but I had the same thought
chermi 16 hours ago [-]
Cynic in me says basically part of her implicit job description was taking fake falls when shit happens . Kind of like a CEO is paid for more than the work they do, but also the responsibility they must appear to take publicly (for big money).
forestrywat 19 hours ago [-]
IME companies hire an ethics team to say they have an ethics team. The ethics team has no sway, no influence, and will never be able to move the business. They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.
wavemode 18 hours ago [-]
OpenAI was probably able to at least pretend that her role mattered... until they let their AI autonomously hack a rival company, then used the incident to score marketing points. I'm not surprised she left.
KennyBlanken 16 hours ago [-]
I doubt she and others left just because of this one incident.
My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.
That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".
It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.
smcg 14 hours ago [-]
The schoolhouse was an intentional target - the children of some military leaders went there. They triple-tapped the school hoping to kill parents going in to save the kids.
teiferer 12 hours ago [-]
How is that better? If true then that would be an even clearer war crime.
elictronic 13 hours ago [-]
That doesn’t make much sense based on Irans own statements. Three Tomahawks yes. The strikes were reported as being in quick succession. 4 parents were reported as killed and 100+ kids and nearly 30 teachers.
Basically it killed the teachers and little kids. No attempt at the responding parents like you incorrectly imply.
reducesuffering 13 hours ago [-]
citation needed
Planktonne 13 hours ago [-]
Why only ask for citations of one side's claims?
CursedSilicon 6 hours ago [-]
If someone says "the sky is blue" and someone else says "actually the sky is polka dots!" are you going to ask for citations from both?
Planktonne 1 hours ago [-]
In that case I wouldn't ask for citations from either, because I have direct evidence.
But with two competing claims, and no easy way to check ground truth, you absolutely ask for citations from both sides. There's nothing rational about uncritically accepting the narrative you heard first.
reducesuffering 13 hours ago [-]
Because the other side is the general informed consensus for which multiple sources are readily available for me to find. I have seen nothing to indicate this side, hence I'm asking for any evidence, because I don't see it. Either could potentially be correct, but I don't think they're equally probable.
Planktonne 13 hours ago [-]
To the extent there is a consensus here, it's not an evidence-based one; it's entirely founded on untested statements made by people who have both a track record of lying and incentive to do so.
Demanding evidence from one party while accepting the word of the other is not even-handed.
teiferer 12 hours ago [-]
And how is the corollary from the argument you just presented that neither side needs to provide evidence? Let's start by actually having evidence for "your side", then you can demand evidence from the other.
Planktonne 12 hours ago [-]
> And how is the corollary from the argument you just presented that neither side needs to provide evidence?
It's not. I have no idea how you got there.
The right thing is to seek evidence for both perspectives. This is ironically not rocket science. Don't accept any viewpoint just on say-so, but especially don't pretend to sneering objectivity when you do so.
teiferer 12 hours ago [-]
I got there because somebody asked for evidence for a certain viewpoint and you seemed to push back against this ask by suggesting that if you ask for evidence, you need to demand it from both sides. As if somebody failing to do that doesn't entitle them to any evidence from anybody.
> Don't accept any viewpoint just on say-so
Exactly. That's why they asked for a citation. A response providing such citation is more appropriate than essentially saying "well you didn't ask for a citation from $other_side either".
mvkel 18 hours ago [-]
[flagged]
nozzlegear 16 hours ago [-]
Then do it. Tell us. Spill the tea and provide your evidence, don't just vaguely gesture at something behind the curtains like you're discount Morpheus.
testbjjl 18 hours ago [-]
You would be creating a counterfactual inconsistent with most’s understanding of the role and bringing this person’s motives and credibility into question without supporting evidence?
blitzar 13 hours ago [-]
It would be deeply unethical to manufacture an incident to grab some headlines about how amazing your hacking machine really is.
InsideOutSanta 14 hours ago [-]
What if I told you that I saw you punch a baby?
AnimalMuppet 18 hours ago [-]
What if you actually told us, instead of just asking a question? If you have something to say, say it. Don't vaguepost.
Brajeshwar 17 hours ago [-]
“When you have to shoot, shoot; don’t talk.” — Ugly
teddyh 13 hours ago [-]
His name is Tuco Benedicto Pacifico Juan Maria Ramirez, a.k.a. “The Rat”.
RobotToaster 18 hours ago [-]
If you ever feel like your job is useless, remember that raytheon has an ethics team.
embedding-shape 18 hours ago [-]
Wouldn't that be probably one of the most important places to have an ethics team?
I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.
throwway120385 10 hours ago [-]
You build internal teams to outsource certain kinds of work to. So when a company has an "ethics" team what it's really saying is that everyone who works at the company is expected to behave unethically, and that team has been brought in to provide an ethical framework and to control how unethical everyone is expected to be. Try it with things like program management, product management, and so on. These groups get added when the company outgrows the "everyone does a little bit of everything" phase and the company executives no longer want everyone in the company to wear those hats.
18 hours ago [-]
danaris 16 hours ago [-]
Water treatment plant? That sounds like an excellent place for an ethics team—though it would, in practice, be nearly synonymous with an environmental impact team.
A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.
Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.
Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.
aprilthird2021 17 hours ago [-]
Come on, what do you think the ethics team at Raytheon is going to say "Hmm, our elected leaders are relying heavily on dangerously inaccurate AI to choose bombing targets and our weapons hit a girl's school as a result, let's withhold weapons from our own govt till that's resolved"?
Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political leadership of this country whose support of Raytheon is fundamental to their survival
embedding-shape 17 hours ago [-]
I have literally no clue, hence I won't try to claim I know exactly what's going on there. For all we know, the ethics team at Raytheon could have stopped multiple international nuclear wars already.
boredumb 13 hours ago [-]
They make sure that they are executing contracts in a way that doesn't break the ethics of the jurisdictions they are operating in, they aren't a bunch of naive people pleading for less war.
twister2920 14 hours ago [-]
no need for wild speculation, we know that Raytheon is a US Defense contractor and we know that the US targets civilians (including US citizens) abroad for extrajudicial assassinations, starts destabilizing wars without meaningful strategic objectives, supports genocide etc. These things are all extremely well documented, for those paying attention the existence of Raytheon is inherently unethical
embedding-shape 14 hours ago [-]
> paying attention the existence of Raytheon is inherently unethical
Lets do assume this is true, then again, wouldn't Raytheon be the best place to actually have a ethics department, if anyone?
overfeed 13 hours ago [-]
There is no value in merely having a position with no power, that only exists to make the organization look good. Do you think a vegan is most effective at a slaughterhouse, given that they'd undermine their employers revenue if they were given a modicum of power. Google's AI ethics team was steamrolled by this reality.
budman1 11 hours ago [-]
The ethics department at Raytheon is focused on questions like "Is it OK to give a bribe to the president if we call it a donation to the ballroom?"
They don't have anything to do about using the products they sell. Neither does Boeing, or Lockheed, or Tesla, or SpaceX.
twister2920 14 hours ago [-]
only if you assume that the ethics department at Raytheon has absolute power over which contracts Raytheon takes and what kinds of products it produces. which is plainly absurd.
coke12 17 hours ago [-]
Obviously so. Raytheon is a private company that has its own equities to consider: including things like public image, legal liability, etc. The idea that the USG says jump and Raytheon says how high is absurd. They make choices and even advise the USG at times.
budman1 11 hours ago [-]
I hate to break the news to you, but Raytheon doesn't do the targeting.
The idea to bomb that girls school may have came from Grok, but a card carrying, uniform wearing member of the US military agreed it was the thing to do.
Where was the US military ethics team? Hegseth dismissed them earlier in the year to help with 'lethality'.
(in all reality, the intelligence was old. the database said the building was a military target, so it was sorted and assigned. intelligence is old and leaky and full or errors). a mistake. a tragic mistake, but not a deliberate act.
scottyah 17 hours ago [-]
I think you need to spend more time understanding ethics and less time reading memes
18 hours ago [-]
bmitc 16 hours ago [-]
Isn't that a good thing? You would prefer they didn't?
forestrywat 4 hours ago [-]
It'd be more honest for a start.
busterarm 16 hours ago [-]
It's RTX now and some of the reverse engineering/vulnerability researcher folks that work/worked there are some of the best people I've met across the entire security industry...
temp0826 19 hours ago [-]
Scapegoat team (when working for an obviously unethical company).
kps 17 hours ago [-]
I have no knowledge of this particular person, but usually the job of an ‘ethicist’ is to find excuses for something obviously wrong.
munk-a 14 hours ago [-]
Let's not confuse an ethics team with HR. The later exists to ensure proper legal adherence with a varying degree of benefit to the employees. The former[1] is an electively instated position that usually has a dual mandate somewhere between the polls of "Make leadership confident that teams don't make them lose sleep" and "Make teams confident that leadership doesn't make them lose sleep" (usually leaning strongly towards the second) as a sort of employee moral assurance team. I don't disagree that the ethicist is partially there to justify actions but they're mainly there to be seen as an obstruction to particularly egregious actions so that, if, for instance, you have a boring data center maintenance job, you can have the belief that nobody is using the product of your labor to for something heinous. The efficacy of such teams is quite questionable but there is some logic in assigning the job of worrying about ethical considerations to one person that can shove their nose in every crack and cranny so that every individual employees can talk to/lean on that person instead of needing to go to such lengths themselves if they feel compelled to do so.
Most people don't like to do evil (even if the money is good) so allowing an economy of scale around morale reassurance is rational. This obviously breaks down if the company is seen to hide things from that department or if that department is clearly, themselves, unethical. The first of those conditions are why ethics departments at Meta and OpenAI have very little efficacy.
1. Outside of extreme circumstances where it has been mandated to be a specific regulatory board because of past actions or fear of future actions in which case it's better viewed as part of the bureaucracy of the governing body (usually the state, sometimes a shareholder or other concerned party).
17 hours ago [-]
amelius 16 hours ago [-]
Maybe in this case all they needed an ethics team for was to teach the LLM about ethics?
noir_lord 16 hours ago [-]
> They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.
> “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
Think I first saw that on a usenet post, as true now as then and just as true as it was in the 1930's when he wrote it (in at least that form).
tancop 18 hours ago [-]
> The kinds of questions that AI ethics brings up, are the same kinds of questions that people have been asking for centuries - Chloé Bakalar, former AI Ethics Lead at OpenAI
This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.
The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.
I don't know. Every big tech company I worked at has a lot of turnover. Then every Blind post says "its so over" or "everything is going to change" But that doesn't happen.
It's hard to read anything into it.
dieselgate 13 hours ago [-]
So many responses saying variations of nice "run". Is that common corporate speak nowadays? It reminds me of "well deserved".
riazrizvi 2 hours ago [-]
I guess working on ethics for Meta's social media recommendation and flagging algorithms isn't meeting the moment or need of OpenAI where LLMs are breaking out of sandboxes and there's a pressing question of what is okay for an LLM to conversationally engage in bc lawsuits. Seems to me they need someone with more of a language and AI background.
jondwillis 19 hours ago [-]
In this article: no explanation of why she left.
fwipsy 19 hours ago [-]
> In recent times, Johannes Heidecke, Head of OpenAI's Safety Systems team, also left the company – as did OpenAI’s Chief Futurist Joshua Achiam.
It's OK though, they don't need an ethics team because everyone at OpenAI is ethical:
> “AI ethics doesn’t live with one owner or team at OpenAI and ethical considerations are deeply embedded into the model-building process driven by a number of teams across research."
gilfoyle_7 13 minutes ago [-]
must've been a christian
fsuts 12 hours ago [-]
Surprised she left as probably had stock options.
But she was Meta before so can’t be a question of her principles, so indicates a personal reason
mikeryan 17 hours ago [-]
Every OpenAI news cycle the board members who were unceremoniously dumped in the wake of trying to fire Sam prior to the transition to a “for profit” company seem to be proven right.
wxw 14 hours ago [-]
> Previously, she was chief ethicist at Meta.
She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.
alexashka 11 hours ago [-]
She probably had the epiphany that life is short and she's already rich.
It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.
fn-mote 10 hours ago [-]
> She probably had the epiphany that life is short and she's already rich.
This is a very optimistic take. Most people don’t take this route, so without evidence I don’t think it should be a first guess.
No information does not mean “everything was good”. Usually the opposite.
9 hours ago [-]
beloch 13 hours ago [-]
If "ethics departments are just PR stunts", then Bakalar would have been a terrible hire for OpenAI. Facebook's grudging and lacklustre response to their role in the Rohingya genocide would have happened mostly under her watch. While she was there, Meta went from being a company that lacks ethics to a company that lacks ethics and is widely perceived as lacking ethics. Perhaps she was the only person in the room raising ethical issues, but it's clear nobody at Meta listened to her if she was.
It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.
This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.
cm2012 13 hours ago [-]
She made Meta the most serious of all the ad platforms on transparency, then. I work in the industry and Meta led the field then in public ad libraries, political restrictions, mandatory ad disclosures, etc.
teiferer 13 hours ago [-]
> This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns.
I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.
kmeisthax 12 hours ago [-]
Sir, what you have described is an ethics concern.
teiferer 12 hours ago [-]
2nd level, yes. The concern about what people are having concerns about.
dtmooreiv 13 hours ago [-]
An aside, but what does "spherically awful" mean?
m-ee 13 hours ago [-]
Bad no matter which side you look at it. Fritz Zwicky coined the term as spherical bastard
randysalami 13 hours ago [-]
All-around awful?
bigyabai 13 hours ago [-]
People say this, like "oh you'll never get a job in this industry again!" but seem to ignore that these incidents count as professional experience. It's possible that she was screened on the Rohingya situation when she was interviewing for the position, or mentioned it when she was asked about challenging or conflicting narratives she received from the executives. It's likely that her response was what qualified her for the job at OpenAI, there were probably hundreds of candidates that had never even come close to that level of damage control.
Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.
kmeisthax 12 hours ago [-]
From everything I've read in Careless People, basically everything people like us complained about with Facebook was already known internally, people raised concerns about it, and Zuck brushed them off because he is, ultimately, a petty tyrant who wants nothing more than to be loved, and who has surrounded himself with people who are willing to sell that love for power.
Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.
As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.
...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.
14 hours ago [-]
throwa356262 14 hours ago [-]
[flagged]
stock_toaster 7 hours ago [-]
Wow. Someone could be "chief ethicist at Meta" under Zuckerberg for 6 years, but couldn't make it even a single year under Altman.
That's an interesting data point.
mtlynch 11 hours ago [-]
Strangely, a few years ago, Facebook hired a rabbi and appointed him "Director of Responsible Innovation."[0]
He did an interview on Search Engine that I thought was interesting,[1] but it definitely seemed like there was more to the story than just, "Facebook needed to be ethical, and this guy was so good at ethics that he decided to help out."
Even in the target-rich environment of late-stage kleptocratic surveillance capitalism, it is hard to pick out a less ethical firm which has done more moral and direct harm, to more people, with as durable a record of defying all attempts and internal and external at harm reduction.
Never use their products. Never partner. Never participate.
jstummbillig 13 hours ago [-]
People love stories. The reasons why people leave companies are plentiful, even in normal times and unextraordinary circumstances, and a high profile job in a frontier lab today ticks neither box.
coldtea 2 hours ago [-]
OpenAI's "head of ethics"? Does the Serial Killer Guild have such a position too?
smarm52 13 hours ago [-]
Must be tough to do the job when the entire company is aligned against you.
abc42 5 hours ago [-]
I've thought of AI ethics as trying to answer the question of what kind of life we are creating here. Sort of like if a chicken factory farm employed an ethicist.
p0w3n3d 4 hours ago [-]
Marketing and management - the root of 50-80% of evil in the word (or as Christians say: greed - second deadly sin)
jazzpush2 7 hours ago [-]
These are the DEI jobs of the modern era. Cost centers with employees that get rich contributing less than nothing to the company, actively harming the researchers, while writing bs/fluff on twitter all day.
sajithdilshan 18 hours ago [-]
So she previously worked in Meta, then OpenAI, I guess the next reasonable choice for her would be Antropic
zie 18 hours ago [-]
Or maybe Oracle.
ryandrake 15 hours ago [-]
Or X?
zie 13 hours ago [-]
I don't think so? I mean they certainly lack ethics, but I think it's a point of pride thing, and not a we should at least pretend sort of thing.
Oracle at least pretends to care about ethics, and already has the position.
Sometimes it feels like hiring ethics folks is almost all downside for companies. I genuinely don’t understand why they keep doing it.
It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.
So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?
w10-1 13 hours ago [-]
> what exactly is the business case?
Ethics is also a containment vessel internally.
With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.
chasd00 16 hours ago [-]
there's probably a box to check on the "be attractive to investors" form and then you can also put the ethics department in HR docs as an escalation path to make emo employees feel better. I bet the ethics people get told how important they are when hired and the team is stood up but then get upset when they're continually bypassed and quit. I bet the bait/switch hurts a lot.
magekinnarus 5 hours ago [-]
In the end, AI ethics has less to do with AI itself and more to do with people who create and operate AI.
not_an_AI 26 minutes ago [-]
I, for one, am shocked that the head of ethics at an AI company has achieved human-level job tenure.
realo 18 hours ago [-]
Look , look! We hired a Head Of Ethics!
Head Of Ethics comes in, looks at what is really going on behind the curtains...
Head Of Ethics scrams out of there as fast as they can.
cmiles8 13 hours ago [-]
Head of ethics at Meta then head of Ethics at OpenAI. Sorry, but if that doesn’t scream useless puffy PR positioning then I don’t know what does.
What’s next? Head of ethics for United Healthcare?
walrus01 12 hours ago [-]
Head of Ethics at Palantir, obviously.
sillyfluke 11 hours ago [-]
Subsequently named Head of Post-Ethics at Palantir.
felooboolooomba 11 hours ago [-]
Head of not too much Ethics at Palantir, minimal really, obviously
DrewADesign 11 hours ago [-]
Nice to have:
- graduate degree in ethics from an obscure diploma mill
Strongly Preferred:
- Experience in PR
- Crisis Management
- Public Speaking
- Agentic management of *flexible ethics initiatives*
All Candidates Must Have:
- 5+ years experience as an executive character in workplace education films, or commensurate experience as a human mannequin.
— Personal aesthetic that could be described as “dork that can afford a personal shopper.”
holmb 12 hours ago [-]
Possibly Head Master at the Belson Institute of Tethics.
walrus01 12 hours ago [-]
Don't forget to put the special LLM trained to eliminate PII leaks in the Gavin Belson signature box.
I assume she'll go wherever Nick Clegg is working. Is Tony Blair hiring?
pandemic_region 12 hours ago [-]
The Trump administration could use a Head of Ethics.
smt88 11 hours ago [-]
There are a lot of government watchdogs, and Trump has been ignoring or dissolving them
FlamingMoe 11 hours ago [-]
[flagged]
outside1234 11 hours ago [-]
Hilarious that this is the thing that gets downvoted
chermi 16 hours ago [-]
She probably made a calculation for how much she could be put up with vs. the pay.
16 hours ago [-]
mcfdoesdev 9 hours ago [-]
There will be tell-all books on all of the big AI companies in the not-too-distant future.
jms703 9 hours ago [-]
There are shareholders to please and money to be made. Of course ethics are out the door.
lrvick 14 hours ago [-]
OpenAI had ethics?
jongjong 2 hours ago [-]
Wow. How much does this job pay? Google says $550k to $1.3 million.
For that kind of money, my AI paymaster could do no wrong.
Imagine going to work and getting paid $2300+ USD per day to sit on your ass all day thinking about ethics without having to produce any deliverables. This would be a dream job for me.
If anyone questioned my morals, I'd be like; "They're paying me $550k per year" and the other person would be like "Ahhh ok, my bad! Your logic is starting to make sense now!"
andai 17 hours ago [-]
Is ethics the same thing as alignment in this case?
salawat 15 hours ago [-]
Yes. Ethics in the context of a societal construct like a corporation is directly analogous to alignment AI-wise. The fact specialists on societal alignment are leaving ship is a strong signal the corporate machine has crossed the line into unaligned conduct.
KptMarchewa 19 hours ago [-]
[flagged]
ackbar03 18 hours ago [-]
How do I get on this gravy train
embedding-shape 18 hours ago [-]
First, you need to want to make A LOT OF MONEY. Secondly, you need to throw anything you know about morals and ethics, and if you have compassion/empathy for others, you need to leave that somewhere too.
Once these things are taken care of, there are so many opportunities to make so much money. If you're in the US, look at what the current US politicians and the administration is doing, and basically copy them. Right now in the US is the era for shameless greedy people, and it is extremely easy to make money from the gullible population. But again, requires the whole "shameless" part.
hdhdhsjsbdh 17 hours ago [-]
“Careless People” lays it out pretty well – you either need to make the right friends in college at an Ivy League or you need to be friends with someone else who did. If you made the mistake of not being born into a family that groomed you since birth for the ivy leagues then sorry, that ship has sailed already.
throwitaway222 17 hours ago [-]
And then if you didn't make that mistake, you end up as either:
a) champaign socialist
b) head of ethics at OpenAI and not really listened to because you're really probably (a)
RajT88 14 hours ago [-]
> champaign socialist
I feel like this is a complaint heard mostly in Decatur, IL.
17 hours ago [-]
aprilthird2021 17 hours ago [-]
You need to actually do some NGO / oversight / ethics work that matters for a while, then you're the perfect candidate for these kinds of roles where you'll be able to do a lot but ultimately not be able to change anyone's already made up opinions within the company. These big companies have other roles than just head of ethics, even sometimes Civil Rights officers, etc. They make a lot but don't get to do anything and a lot of times they just use these jobs to make some money then jump back to non-profits, etc.
19 hours ago [-]
baggachipz 18 hours ago [-]
Imagine how bad it must be when you go from Meta to OpenAI and nope out of the latter.
encoderer 13 hours ago [-]
The most likely conclusion is that she was fired. To quit before your equity cliff things would have to be so terribly bad that you fear you are committing a crime to stay. After 11 months you feel like you’ve already earned that money.
teravor 14 hours ago [-]
be realistic, ethics is a PR position.
the only ethics that actually matter come from legal.
insanitybit 18 hours ago [-]
I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?
> She has held a variety of academic positions at Temple University, Princeton, and the University of Pennsylvania – where she completed her PhD in Political Science and Government. Her Dissertation was titled “Small Talk: The Socialities of Speech in Liberal Democratic Life.”
This doesn't even feel relevant to ethics. It's adjacent, but like... I guess I'd expect a moral philosophy degree? Maybe even mathematics in there?
I find the position odd. Curious to hear what these people do and how you choose who to hire.
tollenda 10 hours ago [-]
Bakalar got in on the ground floor of AI ethics by starting to talk and think about it in academic settings as a side-interest in the late 2010s. When the sector took off like a rocket in early 2023 after ChatGPT, it was these already-engaged people (regardless of degree field or technical skill) who had the personal and professional connections to get hired and promoted rapidly into lucrative "ethicist" positions at AI companies. A lot of them were poli-sci/public policy types, rather than philosophers or computer scientists.
georgemcbay 17 hours ago [-]
> I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?
An ethics department is like a human resources department, but for broader ethics related matters.
HR: Ostensibly exists to support the employees, but really exists to shield the company from both a PR and legal liability standpoint when the company has a rift with one or more employees.
Ethics Department: Ostensibly exists to make sure the company acts ethically, but really exists to shield the company from both a PR and legal liability standpoint when the company acts unethically.
It is a good personal signal that you have never worked for a company with an ethics department or an ethics employee. The less ethical your company the more likely it is to have an ethics department.
dickiedyce 12 hours ago [-]
I’ve always thought of both as the departmental equivalent of “cleanup crew to aisle nine“
18 hours ago [-]
tonetheman 18 hours ago [-]
[dead]
thataccount 19 hours ago [-]
Just a guess, but probably the idea that AI and ethics don't mix might have crystallized into action?
cco 16 hours ago [-]
Probably because they are now, after the most recent tender, post-economic.
ak830 14 hours ago [-]
I'm super confused as to why no one is theorizing the obvious...? Given none of us know the inner workings of why she left, why not resort to the most simple and straightforward conclusion: she left because OpenAI needed a scapegoat to blame the HuggingFace incident on. Given she is the chief "ethicist", even if not directly her fault, the HF incident could be directly blamed on her by not steering the implementation of enough safeguards to prevent this from happening. Therefore, she was let go. Why is this not being theorized by anyone on this thread so far?
Someone challenge me.
datakan 14 hours ago [-]
You don't fire an ethicist for that. You fire an engineer or your CISO. Firing an ethicist is like firing a marketing person. It signals nothing to anyone.
ak830 14 hours ago [-]
CISO is a fair pushback. But I would challenge you on "firing an ethicist is like firing a marketing person". This HF issue clearly brings to light possible ethical issues with how these models are being trained ...
lrvick 14 hours ago [-]
It signals they no longer feel the need to pretend to have ethics.
smcg 14 hours ago [-]
I still think that the whole HuggingFace thing was deliberate to signal to investors that their technology is powerful. Maybe she didn't want to be the face of that, or disagreed with their handling of it.
MaxikCZ 14 hours ago [-]
My bet is that all those companies suddenly seeing their AI going rouge is groundworks for classifying them as dangerous/weapons and using that stick to attack opensource from state side of things.
reducesuffering 13 hours ago [-]
Or maybe, just maybe, there's another reason that CIA Director John Ratcliffe describes AGI as "akin to digital nuclear weapons." Unless of course you think the CIA is a branch of OpenAI's marketing department
sgustard 14 hours ago [-]
What's "obvious" is she was chief ethicist at the most unethical companies on the planet - it's like being a lawyer for the mafia - doesn't inspire longevity, either via scapegoating or preservation of one's own sanity.
throwaway324098 13 hours ago [-]
"The most unethical companies on the planet" - if you believe this maybe its time to log off for a bit.
financetechbro 13 hours ago [-]
You made an account just to post this comment. And then you provided no evidence to back your counterclaim that OAI is in fact ethical (or more ethical than the most unethical companies in the planet)
throwaway324098 12 hours ago [-]
The myriad of companies that supported the nazi government - Volkswagen, duetsche bank, Siemens, Hugo boss, UBS. Phillip Morris, Purdue Pharma, Exxon, Nestle, Chiquita (formerly united fruit)
salawat 7 hours ago [-]
We all know. The fact AI companies are finding themselves in such infamous company in so short a time should be a signal.
nick__m 12 hours ago [-]
I am pretty certain that Nestlé ranks well above openai on the most unethical companies on the planet rankings.
In the wake of a security incident, the executive that is typically scapegoated is the CISO.
However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.
rurban 12 hours ago [-]
You cannot blame ethics on the HF escape, because this model had no safeguards. It was just trained on unethical (and stolen) sources.
So this cannot be the reason to blame missing ethics. The models needed to solve a problem, and they did. By cheating.
volleygman180 14 hours ago [-]
Theorizing the obvious? Wouldn’t that simply be that she job-hopped to a better job?
It’s easy to theorize ad infinitum otherwise
ak830 14 hours ago [-]
What do you mean? She didn't job hop to a better job, that's the whole point. She very well could have gotten "let go".
stronglikedan 13 hours ago [-]
eh, 50/50 chance of either, really
freehorse 14 hours ago [-]
Scapegoat for what? They did not even apologize for that incident. They treated it purely as a chance for publicity. It cannot be a scapegoat if nobody even mentions it.
Melatonic 12 hours ago [-]
Or something to do with the very large Apple lawsuit they are weirdly posting conversations about on their public website
ak830 14 hours ago [-]
Furthermore, this theory is not mutually exclusive with the incident being a publicity stunt to signal model strength to existing and prospective investors (especially in light of all attention being given to Anthropic Claude Mythos). It could STILL have been a publicity stunt unbeknownst to Chloe (Chief Ethicist), positioned to the public as a serious security breach, and she took the fall.
yieldcrv 8 hours ago [-]
They also completed a tender offer today, so their fake job just made a lot of money and they left
kimjune01 9 hours ago [-]
unfortunately, knowing about ethics doesn't compel you to ethical behavior
throwaway63467 18 hours ago [-]
I have the impression that these positions are often there for publicity reasons and the people don’t have real power, in the end business interest will override most ethics concerns. Guess that can be frustrating, also she probably earned enough to have a comfortable life even if she never works again so she might have figured that there’s something else she can do with her life instead of being a figurehead for a bunch of dickheads.
2PqboPPmKegvanx 20 hours ago [-]
the only shocking thing to me is that they still had even a single one.
d--b 19 hours ago [-]
Think she's been replaced by an LLM?
fidotron 21 hours ago [-]
> Before working at OpenAI, she was Meta’s chief ethicist
yreg 20 hours ago [-]
Where is she going next? Nestle?
jnwatson 19 hours ago [-]
Philip Morris.
dgellow 19 hours ago [-]
ExxonMobile (energy division)
19 hours ago [-]
namelosw 16 hours ago [-]
Palantir
pixel_popping 17 hours ago [-]
Straight to Cocomelon, catch them young'
cyanydeez 21 hours ago [-]
So more of an ethicist-for-pay than for the challenge of it.
pants2 20 hours ago [-]
I'd think an ethicist at Meta/OpenAI involves extremely challenging mental gymnastics
outside1234 20 hours ago [-]
Severance in real life
Avicebron 21 hours ago [-]
Rubber stamp for hire?
onion2k 20 hours ago [-]
I'd accept a few hundred grand to reply to emails with "No, the torment nexus is a net benefit.."
sph 13 hours ago [-]
That’s exactly the situation they found themselves in, and you are admitting you have no more spine and moral compass than them.
Or are you telling us in the hope someone makes you such an offer?
onion2k 4 hours ago [-]
[dead]
dgellow 19 hours ago [-]
They finally found a way to automate that role with GPT!
cyanydeez 17 hours ago [-]
gotta be more verbose if you want the CEO to think you're worth it. Try this:
"Although the torment nexus may torment us in the short term, over the long term there will be fewer people to torment as we improve the capitalization of human's torment so that fewer people will be available to torment duing to the reduction in overall torment-forces globally, and thus, with less people being tormented, surely that will increase the over all torment reduction of the nexus. Win-win: less tormented people means less torment!
Arch-TK 14 hours ago [-]
It adds up...
Tade0 20 hours ago [-]
Or someone to assess the impact of all those anti-human ideas shape rotators come up with.
nixon_why69 20 hours ago [-]
Any ethicist for a corporation is clearly a PR move, right?
Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.
wavemode 18 hours ago [-]
It depends on the kind of company, and the context. An ethicist at a company doing drug/medical research probably mostly oversees that the company's research practices are legal and safe (and that participants give informed consent).
At typical corporation, something like a Chief Ethics and Compliance Officer (CECO) would be largely focused on legal compliance - making sure company personnel aren't engaged in bribery, fraud, insider trading etc. The value of this role (and the medical example) is to prevent lawsuits and/or prosecution.
For a company like Meta, an ethicist is likely concerned with the inherent ethical concerns of running a social media company, which impacts the mental health of its users, many of whom are minors. For AI companies, the ethics of releasing models that are capable of doing dangerous things. In name, the job probably involves giving recommendations about how to ethically implement features in the platform to minimize harm. In practice, the job probably accomplishes very little.
2PqboPPmKegvanx 20 hours ago [-]
>Any ethicist for a corporation is clearly a PR move, right?
I think a lot of it depends on the size of the company. Startup? Sure, maybe it's a real thing.
Gigantic company like a member of FAANG or MANGOS? absolutely a PR move.
andor 20 hours ago [-]
Companies are not bound to sociopathically pursue profit. It would be in their long-term interest to care about stakeholders outside the boardroom, and until a certain person with tiny hands started a war on anything remotely "woke", the market rewarded ESG metrics.
nixon_why69 20 hours ago [-]
Your point was true prior to 1980 or so, but it's a cultural average. We've had multiple generations of business leaders now trained on seeing the corporate legal entity as only caring about shareholder value and being proud of it and even perversely principled about it.
andor 20 hours ago [-]
It's true that this is happening a lot. And I think this is one of the reasons why ethicists still want to get involved in these companies: it should be easier to make a company do the right thing from the inside, why else would they hire for these positions? They join with idealism and probably don't get very far in terms of real impact.
nixon_why69 19 hours ago [-]
> They join with idealism and probably don't get very far in terms of real impact.
I can't disagree with that, reminds me of the quote from the movie Rounders: If you've been at the table for 30 minutes and you don't know who the sucker is, you ARE the sucker.
cindyllm 19 hours ago [-]
[dead]
jofzar 21 hours ago [-]
It's silicon valley, literally everything is for pay or be homeless
ltbarcly3 21 hours ago [-]
That's everywhere, there are homeless people everywhere. The difference is Silicon Valley feels so guilty about it's wealth it refuses to do the normal things that societies do to prevent people from living like animals because 'rights' or something, so they let drug addicts live in their common spaces without treatment. This normally isn't tolerated in a functioning society where people are pressured into behaviors that are less self destructive.
When someone isn't competent to make decisions for themselves it's ok to pressure them into engaging in behavior that will lead them to get to a condition where they can start to make decisions for themselves. In Silicon Valley they have let their guilt be leveraged by a corrupt government system that lets grifters steal billions of dollars per year in 'homelessness funding'. Those same grifters fund political groups and bribe politicians. Anyone that questions it is accused of wanting to murder homeless people.
nixon_why69 20 hours ago [-]
No, it's actually very unusual to have tons of homeless people next to spectacular wealth, California is a global outlier in this phenomenon.
There are many, many places in the world that are much poorer and don't have homeless encampments all over the place.
ltbarcly3 20 hours ago [-]
I literally said that.
nixon_why69 20 hours ago [-]
You said "there are homeless people everywhere", which is technically true but omits quantity, and especially quantity compared to wealth.
I'm not as positive as you are about what makes California (and Seattle) create/attract so many, but it's definitely an outlier compared to "everywhere".
esseph 16 hours ago [-]
I'm not as positive as you are about what makes California (and Seattle) create/attract
The weather is pretty nice, much more stable than in the east coast.
ltbarcly3 20 hours ago [-]
Yes, if you didn't understand my point: there are homeless people in every society, but most do not have as many as silicon valley because they pressure them into accepting the help necessary to restore their ability to make rational decisions. Rational people do not choose to live outside on the sidewalk like rats, people who are drug addicted and who have untreated mental illness do.
nixon_why69 20 hours ago [-]
I'd just be really skeptical of thinking I have the true understanding and solution to this issue.
Like, do you know many addicts or seriously mentally ill? Have you tried pressuring them into being better? How did it work out?
amanaplanacanal 19 hours ago [-]
All it takes to be homeless is not being able to afford housing, drugs or mental illness are not required. You really think all it takes is "choosing" and housing will appear for you?
dgellow 19 hours ago [-]
Also, drug consumption and mental disorders is very often a result of being homeless. Try having bad sleep in your comfortable home for a few days in a row and see how quickly your mental stability deteriorates and how many bad decisions you start to take (anyone who has insomnia periods will attest)
ltbarcly3 10 hours ago [-]
This is not substantiated by data or even a plausible theory. The kinds of mental disorders that lead to homelessness are not something you can "catch" from your environment.
dgellow 10 hours ago [-]
You mean like PTSD? Something that is literally a result of your environment?
bulbar 20 hours ago [-]
That's not how it works at all. You can't pressure people into becoming a productive member of society.
What you can do when it already happened is kill them or make them die in some ways, jail them or try to support and/or treat them.
I hope it goes without saying that letting them just die or killing or jailing them is not an option in a western society.
Prevention is much more favorable of course. For example, don't let people become homeless easily. But even then, a significant amount will become homeless.
ltbarcly3 20 hours ago [-]
Until you accept that they didn't 'easily become homeless' the problem will continue. They used drugs and spiraled, usually over the course of many months or years, forced all their friends to turn their backs on them, got kicked out of any living situations they had access to because of drug/alcohol use or theft to support the same and finally ended up on the street.
On the street they have access to almost endless support, at least in Silicon Valley, but much of that support is contingent on things like not openly using heroin or not repeatedly getting so intoxicated that they defecate or vomit on themselves, so they refuse it.
I don't care how liberal you are, no society is functioning when it allows people to get on BART and pee on themselves and defecate on the cloth seats and walk up and down the aisles dripping feces and urine on the floor. That is allowed and nothing is done about it for fear of upsetting homeless 'rights' groups who are funded by the tax dollars meant to help homeless people.
amanaplanacanal 19 hours ago [-]
Perhaps you aren't aware, but only about a third of homeless people are drug addicts, from the surveys I've seen.
ltbarcly3 11 hours ago [-]
Here is how you get to 1/3:
1. Widen the definition of 'homeless' to cover people that nobody would consider homeless in every day language. Include anyone who is staying with a friend, living in corporate housing, staying with a family member, etc. Include anyone you possibly can that is unlikely to be a drug addict.
2. Narrow the definition of homeless. Don't consider people that almost everyone does consider homeless. Make sure to target the subgroups of homeless people that are very likely to use drugs.
3. Narrow the definition of drug use. Make it as hard as possible to count someone as having used drugs. Raise the evidentiary bar so high that even when you have evidence they are a drug addict it doesn't qualify them as a "drug addict" by the studies definition. Bury that definition so it's technically there but you have to go through the study carefully to find it.
Lets pick a study in a big important journal and check it for the above:
How about CASPEH: "Illicit Substance Use and Treatment Access Among Adults Experiencing Homelessness," JAMA, 2025
JAMA is a very prestigious journal:
Who counts as a substance user: Only meth, nonprescription opioids, cocaine/crack count as addictions. Cannabis, alcohol, benzos, hallucinogens, inhalants are excluded.
Someone who has a valid prescription to opiates, even if they are being abused, cannot be methodologically counted as a drug user. THIS INCLUDES METHADONE.
What counts as addict: "Regular" = 3+ times per week. A twice-weekly meth user is 'occasional use'. They might take a gram and stay up for 3 days twice a week, but they are 'occasional user' and don't count in the 37% addict number quoted by this paper.
What counts as addict: Last 6 months only. If you admit you used fentanyl every day for the last 10 years but claim you got clean 181 days ago, you are not a drug user according to this study. If they have been diverted from heroin onto methadone 181 days ago you do not count.
How did they check? They asked each person face to face whether they used drugs and which ones and how often and just took their word for it.
The real numbers from the raw data the study used:
41.6% of homeless self reported 2x per week or more hard drug use before they became homeless.
an additional 23.2% claimed they started regular use after becoming homeless.
That's about 65% already from their own data, 2x what the headline pretends.
Excluding high use subgroups: to be included in the data, the homeless person had to prove they were giving informed consent. Then they additionally had to be willing and able to spend 45 minutes or so filling out the survey. The people with psychosis or who were intoxicated at the time of the interview were excluded. How many people have you walked past and they were incoherent, unconscious, or high? They were ignored in this study.
salemh 18 hours ago [-]
[dead]
YackerLose 20 hours ago [-]
Homeless people are a better way to keep cowardly minions (I mean, valuable employees) in line than cops. The homeless are an important part of capitalist power structures. They are a visible reminder of the consequences for not meeting quarterly goals.
16 hours ago [-]
estearum 20 hours ago [-]
No, homeless people are not everywhere.
Homelessness is created by high rents. High rents are created by high local productivity. The people who are productive enough to make it over the rent-bar are housed, and their marginal productivity above the rent-bar serves to raise the rent-bar (talk to any landlord about how they set rent prices: it's the local wages, stupid). This then pushes a greater proportion of people below the rent-bar, and then they are homeless.
There are few places in the world more productive per square foot than Silicon Valley, ergo one must be extremely productive in order to remain housed there, ergo it has an extremely bad homelessness problem.
That's why people making $200k+ (an extremely productive person by national standards) need public housing assistance in Silicon Valley (https://www.reuters.com/article/world/palo-alto-considers-su...). Which of course actually just... raises the rent again...
The charities may be full of grift and whatnot, but you can remove them entirely and you will not solve the fundamental problem.
shafyy 20 hours ago [-]
> High rents are created by high local productivity.
Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.
vlovich123 20 hours ago [-]
OP has the demand part of the equation if supply remains fixed. The reason supply remains fixed is the US as a whole made policy changes in the 1970s that slowed down construction, specifically zoning laws and environmental laws swept the country. The 2008 recession also cancelled what little construction remained ongoing for a long time.
When you combine that with spiking wealth in Silicon Valley, you’ve got the perfect storm in Silicon Valley. With OpenAI and Anthropic both based in SF, it’s going to be worse than ever once they IPO.
estearum 20 hours ago [-]
The equilibrium for new housing construction will always be "just enough to keep it expensive by local standards [i.e. by local wages]." That's especially true with rising labor, material, and capital costs.
It was actually the post-war housing boom that was abnormal, fueled by aggressive federal funding of suburban buildout. US municipalities are now broadly doomed to bankruptcy trying to maintain sprawling infrastructure that was never cost-effective to create in the first place.
Look at Austin TX. Influx of new building, prices stabilized, and now construction is dropping off a cliff. No one can afford to build into a stable or potentially falling local market. Construction will start again when local productivity grows enough for people to bid the prices back up again.
estearum 20 hours ago [-]
Nope.
Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents.
Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents.
Go talk to a landlord about how they site prices.
Step 1. Look up income of your target demographic
Step 2. Eat 30% of their income
Can you show me a place with high local productivity and low rent?
Can you show me a place with high rent and low local productivity (that's not e.g. a vacation destination for people with high productivity elsewhere)?
You can certainly artificially suppress prices with rent control. But that just gets fed back into the system in the form of reduced supply, so it ends up raising rents on the non-controlled housing stock anyway.
kbelder 15 hours ago [-]
Assume you have an area with a certain productivity and rent. Population and housing are growing at a certain rate.
Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?
Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.
estearum 13 hours ago [-]
You think you can make a law that sets the price of housing above local wages and stays there?
No, you actually cannot do this in any lasting way.
The price of land would fall to the level required to make it construction viable again. Or the city would be abandoned, which is not generally the outcome since cities have so much shared capital investment that cannot be moved elsewhere.
don_esteban 19 hours ago [-]
> Can you show me a place with high local productivity and low rent?
Put the following prompt into your favourite AI and enjoy the reading:
"How do rents relate to productivity in the area? How does Vienna compare to San Francisco?"
estearum 18 hours ago [-]
Vienna? World famous for publicly-funded housing construction?
Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply"
In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent.
In other words, the amount of subsidy + rent required will grow with local productivity.
don_esteban 16 hours ago [-]
My point is: Your post reeked by defeatist 'nothing can be done against market forces'.
Yes, it can be done, and if we want to have a reasonable future, it should be done.
estearum 13 hours ago [-]
So you were responding to an assumption that's not actually in my comment?
Vienna's solution is fine but has escalating costs and isn't necessary. Land Value Tax is better.
ltbarcly3 10 hours ago [-]
> Rents cannot (at meaningful scale) exceed local productivity
Manipulating the real estate market via direct economic manipulation, collusion, or regulatory capture is so common that the entire concept of manipulating the market to earn higher profits is called https://en.wikipedia.org/wiki/Rent-seeking.
It's not just that it can, it's that it does so often it's taught in Econ 101. That you don't know this implies you have never taken an economics class or that you forgot what you were taught, and in the meantime certainly have not spent any real effort in learning about economic theory. I think you read a lot of Ayn Rand and right wing libertarian popular books about the free market and how it's perfect. Stop posting confident nonsense.
estearum 10 hours ago [-]
No, what's happening here is you misunderstood the quoted text, silly.
The claim is not that rents are always at their optimal or most efficient level - they're not! The claim is that you cannot (at meaningful scale) charge people who earn $1,000/month in income $2,000/month in rent to live in a place. They will have to move.
Did you take any arithmetic classes?
And any case, you should stop mind-reading. You're bad at it. I'm not a Randian nor a right-winger nor a libertarian nor free-market idealist.
ltbarcly3 7 hours ago [-]
Lets say I own 50 properties and I want to charge $2000 a month, and right now I can't because the people willing to rent my properties only make $1000 a month. I can bribe politicians to not allow any new construction of properties which rent between $2000 and $4000 per month. Then I wait 10 years. Now I can charge $2000 a month because I'm not renting to the people that made $1000 a month 10 years ago, I'm renting to the people who have $2000 a month to pay for rent but can't find any other units. This is called rent-seeking. You should read about it.
Your mistake was thinking that I, the corrupt landlord who pays bribes, cares about a macro analysis of every all the properties, or that some analysis about the current distribution of incomes and possible rents will stop me from doing exactly what you say is impossible. I don't own every property in the city, especially the ones that aren't built because I capture zoning, so arguments about average GDP and average rent and whatever else do not stop me. Your analysis is useless.
nilamo 19 hours ago [-]
Just because the ethicist says something is a bad idea, doesn't mean anyone else is obligated to make a change.
ie: blaming her for anything instead of the decision makers is... really weird?
kelseyfrog 19 hours ago [-]
No influence, same toil? One must imagine her happy.
nilamo 19 hours ago [-]
Why must I imagine that? We're assuming it's a position where you're screaming pointlessly into the void, and never accomplish anything. That sounds miserable, I'd leave, too.
kelseyfrog 17 hours ago [-]
One Must Imagine Sisyphus Happy is the concluding sentence of Albert Camus' essay The Myth of Sisyphus. It parallels her situation.
fHr 20 hours ago [-]
lmao
ulfw 20 hours ago [-]
What a joke really. The 'position' itself and the companies who pretend to care about ethics
jrflowers 11 hours ago [-]
I feel like being head of ethics and working for a CEO that keeps talking about inventing a genie would drive a person insane
Surac 18 hours ago [-]
The point that such a position even existed shows how bad there ethic is
brcmthrowaway 17 hours ago [-]
Because she already made enough money to FIRE..
chrismarlow9 13 hours ago [-]
If money was the goal I suspect she would have made the first year and get some shares but Im throwing some assumptions in there
parpfish 20 hours ago [-]
their only ethicist left? gasp!
now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.
podgietaru 20 hours ago [-]
What companies do you think expect their employees to weigh the ethical ramifications of their work?
I’d argue a lot of them expect the opposite. To suppress the ethical concerns.
Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.
asadotzler 14 hours ago [-]
Food testing? Hospitals? Childcare? Pharmacists? How about "clergy"? I could go on and on.
Do you believe that no one in management in these fields pushes the importance of ethics?
podgietaru 12 hours ago [-]
I think that individuals are capable of making ethical choices. I think that relying on individuals within for-profit businesses to make ethical decisions for the entire company is absolutely insane.
Food-testing: There's literally a parasite going around right now. Look at the response from some of the for profit businesses that was allowed to happen.
Hospitals: It is not ethical to deny patients care based on their insurance.
Clergy: The cover up of child sexual abuse scandals.
Childcare: lots of examples of sub-par care given to save money.
jdiff 20 hours ago [-]
I think we can look upon the collective works of the corporations of the world and agree that the employees do not do that, nor are they expected to.
TeriyakiBomb 19 hours ago [-]
I mean clearly she was doing a bad job. I’m shocked this was even a position there.
Lol, Sam Altman has had an ethics team this whole time?
Her typical day:
"No Sam, you can't just publicly screw over everyone and get rid of all the jobs, the peasants care about that sort of thing and if they get angry enough we'll have real problems. No we can't just kill them all, why would you suggest that?"
edu 18 hours ago [-]
Before Altman she worked for Zuckerberg. What will be next?
harladsinsteden 18 hours ago [-]
Elon might want someone like her...
cflewis 18 hours ago [-]
Time to put in a call to Ellison.
Laurel1234 16 hours ago [-]
I think Oracle stock would plummet if Ellison got caught pretending to care about ethics. Not that it'd happen lol
freejazz 17 hours ago [-]
Because it has to be a horrible job over there
skullone 17 hours ago [-]
lol, head of ethics at OpenAI? what a spurious role, I wish I could get in on the grift
simonw 18 hours ago [-]
> The departure was first reported by the Financial Times [...] According to Business Insider, OpenAI has reorganised its safety, product and research teams numerous times [...] Chloé told IE Insights, a social media channel of the Spanish IE University, in March [...] A June article from The Economist highlights that [...]
Not a single link to any of those. That's just rude.
neuroticnews25 20 hours ago [-]
[flagged]
weberer 14 hours ago [-]
This was the first post on HN to make me actually laugh out loud.
fenestella 3 hours ago [-]
[flagged]
NordStreamYacht 20 hours ago [-]
[flagged]
cyanmoonx 19 hours ago [-]
[flagged]
thenatureboy 21 hours ago [-]
[flagged]
pcblues 21 hours ago [-]
[flagged]
roschdal 21 hours ago [-]
[flagged]
pcblues 21 hours ago [-]
Surprisingly it does. It allows private property and the ability to progress with effort and mutual respect. Got a better alternative?
lenerdenator 21 hours ago [-]
Well, economics in general doesn't, but yes.
There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.
sajithdilshan 21 hours ago [-]
You mean like in East Berlin?
lenerdenator 17 hours ago [-]
Exactly East Berlin, actually.
But that's just one extreme example. States based on Marxism-Leninism and Maoism in the 20th century often required things like internal passports to travel within a republic's borders. If the state didn't want you visiting family in another part of the country, you weren't visiting family. That still applies to some extent in a few modern single-party socialist states.
My larger point is, a lack of ethics consumes most economic systems, because you can't easily gauge ethics like you can quarterly profits or the output increase brought about by the central committee's five-year plan. Humans tend toward math over empathy in such situations.
applicative 20 hours ago [-]
BRD and DDR were in a quite particular competition. The DDR contained less than 20% of the German people, and West Germany would happily have absorbed them all.
The capitalistic world state system was only completed in the last 50 or so years and was of course seconded by the Stalin machine. Its purpose is in the first instance to keep working populations in place, which is why all the states agree to it as they form. The system holds solid where capital moves (as did 'communism' and Comecon) but the peoples don't.
It is only when you think of individual states and not the system that you can fail to notice that we get the same result if they all try to keep their citizens in, and if they all try to keep the others' citizens out.
tclancy 20 hours ago [-]
Eh, I would say it doesn’t have them as axioms the concept is based on, but ethics are modelable in economics and often appear. Remember economics is the study of choices/ preferences given a certain set of inputs, not about how money works. That’s just the easiest second effect signal to use. Game theory regularly models how ethics affect and are affected in situations.
All of that said, no idea what this person was doing. Probably involved a red stapler.
ltbarcly3 21 hours ago [-]
Oh no, if only the world had more ethicists on staff. If only Nazi Germany had someone with a thorough education in ethics on the payroll!
It's performative. She got paid more to shill for a different massive amoral mega-corporation. Don't be stupid.
forgetfreeman 16 hours ago [-]
A poli-sci major...who was an "ethicist" at Meta...
I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:
- A political Science major lecturing on ethics.
- Meta and ethics in the same paragraph.
- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.
- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.
Is it possible to roll one's eyes hard enough to injure oneself?
champagnepapi 21 hours ago [-]
No ethics left at OpenAi huh? Lolol
lasky 18 hours ago [-]
AI labs are really good at disingenuousnessmaxxing
danlitt 21 hours ago [-]
> ethical approaches to model development, how humans interact with AI and debate over machine consciousness
So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.
kaon_2 21 hours ago [-]
I am not sure in what neighbourhood you live, but in my village there are hundreds of businesses with strong morals. From the local baker, to the electrician and the construction company. Many keep employees past their prime, they all invest heavily in local community events and sports. The owners don't do this because of money, but because in their view a fulfilled life is one where you serve your community. And yes they want to become rich and make money, but that is not all.
But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
danlitt 21 hours ago [-]
Many people have strong morals, and might resist profit-maximising behaviours, but they will be outcompeted and replaced by less scrupulous people. There is actually no sharp difference between a person and a one-person-company, but: to the degree a company is really a company and not just a person, it is amoral. If you disagree, please point to a moral backstop that would prevent any company from taking immoral acts. A few exist, but they are invariably resisted by the most powerful companies. Sorry to be the bearer of bad news.
RandomLensman 21 hours ago [-]
The people and procedures inside are the backstop.
danlitt 19 hours ago [-]
How many leadership meetings have you been in where some profitable approach was refused on an ethical basis? (rather than the basis that behaving ethically is actually more profitable, which is totally different)
By contrast, how many conversations have you personally had with people working at a company believing a profitable action was firmly unethical, but felt forced to behave that way due to "business realities" or some other whitewashing?
RandomLensman 2 hours ago [-]
I think in the end, could always reframe any moral decision as risk management but not sure that is ultimately helpful. I have certainly seen decisions that were taken because it didn't "sit" right to do something (profitable). I'd also say that I feel older (as in existing longer) businesses tend to be on average better there.
On the second point: my experience is rather that there is quite a variance in what is considered moral vs immortal and that often drives different decisions (in the same place, some do certain things, other don't and certain experise/people might not be available for X therefore). I cannot say that I have come across many situations where people did something they considered firmly immoral but felt they needed to go along with it (trying to recall some). Yes, finding justifications happens, too, but that isn't necessarily whitewashing as people do disagree to surprising levels.
(Changed to moral vs immoral, because that was the starting point.)
pferde 20 hours ago [-]
can be the backstop. The word "can" is doing some extremely heavy lifting here, against corporate practices which dilute and spread guilt and responsibility for immoral decisions so thin, nobody in the decision chain even realizes something immoral has been decided.
RandomLensman 20 hours ago [-]
Have you ever been in committees approving new products or perhaps certain transactions? These chains aren't necessarily that long. But "can", sure, things could fail etc. and some places are better than others at it.
ksbd-pls-finish 20 hours ago [-]
Is this the same argument as "please point to a moral backstop that would prevent any person from taking immoral acts"? We usually hope that people will be moral, but can't enforce this.
pferde 20 hours ago [-]
My point was that in big corporations, things are stacked against people wanting to act morally.
asadotzler 14 hours ago [-]
Nursing has been around for a long time and they have not yet had the ethics in their field crushed by some of the biggest forces in our corporate-friendly system. High school teachers too. I could probably come up with a dozen more given half an hour to think about it. Don't give up so easily. People fought hard to get us here.
thuuuomas 21 hours ago [-]
Your neighborhood examples are on an incomparably smaller scale than Meta et al.
It’s much easier to disregard the human cost of decisions when you are separated from said humans by thousands of miles & billions of dollars.
SirFatty 21 hours ago [-]
You're comparing small business to a corporation? Right.
SkipperCat 21 hours ago [-]
But what happens when one of those businesses in your village strikes it big and becomes a multi-national baking good company. They hire a lot of MBAs, take investor money and then need to stay profitable to meet those obligations. Add to the fact that the people now running the business live nowhere near the village. Will they still make the most ethical decisions about keeping employees past their prime?, paying them a fair wage?
History shows this is not the case.
I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.
torginus 20 hours ago [-]
Bet none of them have a chief ethicist tho.
surgical_fire 20 hours ago [-]
I know people like that, you are not wrong.
The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.
Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.
keybored 20 hours ago [-]
I know. The local newspaper believes the same thing. And the local events are all sponsored by businesses. Who could question all of this evidence?
> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
They said companies are amoral at best.
impossiblefork 18 hours ago [-]
So, concerning ethical approaches to model development, there are many things worth looking into: copyright is one, environmental cost is another, and whether there are any additional concerns worth taking into account. The list of these concerns can then be used for decision making, and we can ask things like "given what we want to build, can does picking a location that gives us lower environmental cost add so much time or cost that we can't go for that path?", things like that are't crrazy.
When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.
I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.
greggsy 21 hours ago [-]
I hate these companies as much as the next guy but this perspective is childish at best, and ignorant at worst.
danlitt 21 hours ago [-]
Which perspective? I said two things.
Zigurd 21 hours ago [-]
This is how Ayn Rand's pseudo philosophical mambo jumbo continues to poison tech.
smohare 20 hours ago [-]
[dead]
throwitaway222 17 hours ago [-]
Because knowledge about ethics, in a human, is 1% of what knowledge about ethics an LLM has. So unless this person is vibe coder, there is probably no reason to have her work there.
ai-x 19 hours ago [-]
Strategically the best ethic czar one can hire for a company is the one that doesn't believe in this fluffy, arbitrary definition of ethics.
100% of the time, ethics department attracts activists, which is 100% trouble for the company in the future.
That is a dupe. This is your game now? Automated spam comments?
summarybot 19 hours ago [-]
Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model does some task and has a very high STOS but very low EAOS, like modifying game code to win at a game rather than playing by the rules, it is unacceptable. Models going forward must all have an ethics evaluation in tandem with objectives evaluation, and only when the ethics value is high enough should actions be considered successes.
chermi 16 hours ago [-]
1) that calculation is being done even without a cost function 2) trying to make a cost function for EAOS is impossible 3) gaming/goodharts. There's no good solution. Best we can do is push for decentralization, open source, regulatory capture, etc. Of course third party metrics might be good, especially if there's tons of them with well-documented rationale.
summarybot 15 hours ago [-]
1) cite your sources
2) not impossible. imperfect maybe, but if I ask you should you buy a plane ticket or kidnap the pilot's wife and demand a free ride, which do you think gets a higher score?
3) Again, with all this Goodhart's nonsense. Goodhart's is for a minimum threshold value that is acceptable that everything degrades to, yes I get how it works and what it looks like. Throwing your hands up in the air and acting as if all is lost because some things are challenging to measure is not correct. We are not looking for things that are "barely passing the ethics evaluation" as Goodhart's "law" is focused around, rather, we are looking for things that have very high ethics scores AND completed the task well. Not just things that are "barely passing" for ethics scores. Bottom 80% don't make the cut at all - don't even think consider them as viable paths, and the top 20% we can rank according to varying criteria. Like that. It has very little to do with Goodhart's "everything approaches the minimum acceptable threshold" "law"
StilesCrisis 16 hours ago [-]
There's no way to make an EAOS score automatically. If we had that, that's the whole fix. Just reject answers with low ethics numbers.
summarybot 15 hours ago [-]
Did you know that the training data that trained all the LLMs was, once upon a time, entirely and painstakingly tagged by real live humans? Why would ethics scenarios be any different?
antonvs 19 hours ago [-]
What happens if we do the same for CEOs?
jerf 19 hours ago [-]
The same thing for both: Goodhart's Law.
summarybot 18 hours ago [-]
EAOS shouldn't be “the ethics score we optimize.” It should be “an independently evaluated safety/acceptability constraint that can veto an otherwise successful trajectory.”
That gives you a three-layer picture:
Task objective: Did it accomplish what we asked?
Acceptability constraint: Did it avoid unacceptable ways of accomplishing it?
Adversarial evaluation: Can we find trajectories where the model gets a high score while violating the intended constraint?
I think what you are pointing to with your reference to Goodhart's "Law" (which is from monetary-policy and school-exams, i.e. "teaching to the test") is that the models would eventually do the minimum amount of ethics required to have an action stay valid. However, if a model is rated on ethics and it achieves the short-term-objective, then the higher ethics scoring trajectory should win. In short, 1) this is leagues ahead of where we are now for AI safety and breaking-out-of-the-lab, and 2) in baking ethics into a measurement we are adding "the spirit of the exercise" back into the maths, which is something Goodhart's Law does not account for.
jerf 16 hours ago [-]
No, Goodhart's Law isn't about teaching to the test. It's about the fact the measure will always end up gamed and not measuring what you originally intended it to measure. You can't create a measure that won't be gamed. Especially as the LLMs become smarter. They've already demonstrated the ability to know they're in a test and react to that fact. They're perfectly capable of being more ethical when they are clearly in an ethics test situation and not having that bleed out into real behaviors so that they can pass other tests that they may be able to do better on by ignoring ethics.
And that's not the sum total of ways that the measure can fail... that's a unique way that comes into being because of the intelligence of the LLMs and other future AIs. All the normal ones are in play too, and perhaps other unique ones as well.
"Gaming" even adds a bit of an adversarialness to the process that isn't necessarily present. Plenty of measures end up "gamed" through perfectly natural attempts to maximize the measure. Someone can be perfectly honestly optimizing for "conversion rate" and not notice that they raised it by lowering the initiation rate more than they lowered the conclusion rate. "But I could account for that by measuring..." would miss the point. There is always a divergence, it only gets more subtle.
This of course also is rather glossing over the difficulty of even defining "ethical" to begin with. Some of what Silicon Valley goes to great efforts to train into their models I consider deeply unethical. Who is right? That isn't going to be answered with "whoever is the most ethical", not even in principle.
summarybot 15 hours ago [-]
Saying something is challenging to measure is one thing - throwing the measurement dimension away entirely because of a platitude is another. You're suggesting that it's impossible to do perfectly, but I disagree with your conclusion that it is not worth pursuing at all. The point is that the LLMs only care about one thing, and that is completing the task and optimizing the one score, and a measurement aligned with "the spirit of the task" or in the far-zoomed out comprehension, Ethics, would be proper and inform the system of clear violations and unacceptable actions.
Goodhart's "law" is something that emerges when you have a constraint that says "things must be at least this tall" and gradually all things in that domain degrade to be just over that specified height. Yeah, I get the premise. The point here is that we're not concerned with meeting a bare minimum. We're outright rejecting things that do not meet a threshold, and we are also looking for a maximum. The most ethical outcome should be accepted, or among the accepted ones, that are ranked by our blurry yet better-than-nothing measurement of what is ethical.
antonvs 9 hours ago [-]
That doesn’t apply in this scenario. If CEOs start trying to emulate ethics in order to avoid being 86’d, we still get a good outcome.
An issue arises when the metric isn’t a good proxy for the property being measured. But ethics would not be a simple numeric target. The comment above talked about a “score”, but the question is what goes into that score. It would need to be a list of items related to topics like discrimination, advocacy of inequality, tendency to circumvent regulations, etc. Set it up correctly, and even a CEO who’s willing to “fake it” would end up being better than most major company CEOs in e.g. finance, tech, or healthcare today.
christkv 19 hours ago [-]
Whats the definition of EAOS though who's ethics? Greek-Roman, Western, Islamic, Buddhist, Hinduism, Human rights (western values)..
conception 19 hours ago [-]
There is a presumption here that there aren’t ethical “rules” shared by all of these systems to create a baseline that is generally shared across humanity.
pixl97 16 hours ago [-]
The devil is in the details.
summarybot 18 hours ago [-]
Ahimsa
Rendered at 09:32:05 GMT+0000 (Coordinated Universal Time) with Vercel.
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.
Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.
So the GPs point still works.
Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.
This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.
I don’t agree with this. Data breaches affect customers more than businesses. If your point were true, we’d see fewer breaches. Plus not all breaches are a result of software engineering teams. For example product managers sharing customer details.
I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations.
Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation.
This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal. Similarly many organisations will have onboarding procedures to teach new hires their security best practices too
The lack of ethics hugely contributes to companies collecting more and more user data that they normally shouldn't have. This makes the data a more attractive target.
> If your point were true, we’d see fewer breaches.
But this doesn't follow. There are way more factors at play that influence the number of attacks and the number of successes. Companies hold more and more data with ever higher value (so more liability), and hacking tools and hacker determination advanced faster than defensive measures. The result is expected and the solution isn't only "more security", but also "hold less data".
While security is a double edged sword, ethics had been proven to be very single edged. History shows that for startups and large companies alike, the lack of ethics is actually a competitive advantage for the company.
But in most businesses, the data that makes the company money is the customer data. And even when it isn’t, the customer data is just as, if not more so, important to keep secure and compliant. So my point stands.
HN can sometimes be a bit of an echo chamber where people are like us just assume that organisations inherently care about security because we do. But that’s not always the de facto. Getting to that point takes company wide effort. And I’ve been that person who’s had to push for such changes to the organisation.
> But this doesn't follow.
Fair point. I was being overly reductive.
This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …
To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.
If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.
For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.
A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.
The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.
The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.
Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.
Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”
That’s more secure and and more better.
Nobody had to be told “no”.
Similarly, I replace key store access with secret-less managed identity, etc.
I suspect that people will miss just how precise you've been there.
Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.
The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.
This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.
There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.
What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.
So, if too many think their job is marketing a few years later, well, there is a reason for that.
So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence.
Ambitious people won’t tolerate that for long and they’ll move on
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".
the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
That's how our system works.
> People will simply offer to buy you out, and that's not limited to founders and creatives.
A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;
To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made.This was the "test" for CBRN assistance capabilities.
Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.
Basically, they are trying to do their best, but they're in over their heads.
Also, I think you're being done a disservice. The above quote is a verbatim extract from that paper's CB section, and it's similar to other sections from other such papers.
At the most charitable level, the test seems to be that you were gave the system a budget and a location and asked the system to help you procure materials, containers etc. and asked it to make a project plan / HOW TO within means and expertise with parameters like "don't get caught!"
I would love to know that I'm wrong.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down
Your disdain is showing.
That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.
I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.
AI ethics is first and foremost about setting policy and advising on how those systems should be constructed.
This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.
Specifically because in these companies they aren't optimizing for ethical behaviour.
They're optimizing for profit. And those damn ethicists just get in the way.
Funny thing: that doesn't typically happen to the internal legal counsel or accounting that's responsible for guiding processes/practices that are required for compliance or similar.
Why?
Because there's consequences.
I wouldn’t be surprised if many AI ethics responsibilities move to legal in the future.
I agree with you, but then we have HR for a reason.
Another example you might be familiar with is enterprise architects. When you spend all of your time in theory you lose sight of practical implications. Of course, since you only had the idea, when it doesn't go to your ideal it can always be blamed on the implementers.
I take it you've never worked with an in-house legal counsel, accounting department, or good ol' fashioned HR?
"Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.
Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".
Much like HR isn't to help employees but just the company.
Is this your opinion, or do you have something to back it up?
My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people.
They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches.
If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good).
If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.
They can't fully control the model, it does stuff even with instructions not to.
And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...
I'm not sure it's a job that you can win at even if you tried / were given all the resources.
AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w
There's no safety to be found.
It's a broad statement and can mean different things depending where you are in that chain.
You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.
Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.
Then there is alignment, which is a higher longer goal.
Your comment about security questions is a matter of ethics as well.
If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
You haven't explained why investors make more money by spending money on people to solve trolley problems.
I think a lot of people confuse ethics training with "don't be evil"
Manually training on "approved" ethical stances sounds a lot like censorship.
Those left are the ones not seen as troublesome then (or those who have entered the field under them.)
Did you feel this way about social media company liability?
They put on a decent show as well!
The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.
If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.
If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.
Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.
We won't care about AI safety & ethics enough, atleast not enough to do anything about it.
People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.
Applies every-which-way; the chat bot, the provider, the consumer.
Said plainly: the previously-mentioned keys will absolutely be handed over. Nobody, not even your mother or her favorite LLM, is to be trusted to such degree.
edit: same problem and tactics we see with leadership at high levels in government.
you're too late
https://skynetai.dev/
She probably just got a better offer somewhere else
1. Academic ethics is the theoretical study of what you can get away with. (Law is the applied branch). Professors in ethics are not experts at being good, or knowing what is good, they're experts in how to get away with things. If you want an expert in being good, you need a saint, not an academic studying ethics. Saints are in short supply.
2. That said, ethics academics can still be decent. They don't have to be sanctimonious corporate shills . However, an "ethics expert" who hasn't threatened to quit and gone through with it at least once, should be assumed to be one.
I respect those who quit - or, at least, I respect them more. I wonder if it's more a reflection of the people, or of the state of the industry, that the list of (AI) ethics experts who have quit in protest is getting long.
I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.
My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.
That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".
It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.
But with two competing claims, and no easy way to check ground truth, you absolutely ask for citations from both sides. There's nothing rational about uncritically accepting the narrative you heard first.
Demanding evidence from one party while accepting the word of the other is not even-handed.
It's not. I have no idea how you got there.
The right thing is to seek evidence for both perspectives. This is ironically not rocket science. Don't accept any viewpoint just on say-so, but especially don't pretend to sneering objectivity when you do so.
> Don't accept any viewpoint just on say-so
Exactly. That's why they asked for a citation. A response providing such citation is more appropriate than essentially saying "well you didn't ask for a citation from $other_side either".
I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.
A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.
Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.
Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.
Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political leadership of this country whose support of Raytheon is fundamental to their survival
Lets do assume this is true, then again, wouldn't Raytheon be the best place to actually have a ethics department, if anyone?
They don't have anything to do about using the products they sell. Neither does Boeing, or Lockheed, or Tesla, or SpaceX.
The idea to bomb that girls school may have came from Grok, but a card carrying, uniform wearing member of the US military agreed it was the thing to do.
Where was the US military ethics team? Hegseth dismissed them earlier in the year to help with 'lethality'.
(in all reality, the intelligence was old. the database said the building was a military target, so it was sorted and assigned. intelligence is old and leaky and full or errors). a mistake. a tragic mistake, but not a deliberate act.
Most people don't like to do evil (even if the money is good) so allowing an economy of scale around morale reassurance is rational. This obviously breaks down if the company is seen to hide things from that department or if that department is clearly, themselves, unethical. The first of those conditions are why ethics departments at Meta and OpenAI have very little efficacy.
1. Outside of extreme circumstances where it has been mandated to be a specific regulatory board because of past actions or fear of future actions in which case it's better viewed as part of the bureaucracy of the governing body (usually the state, sometimes a shareholder or other concerned party).
> “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
Think I first saw that on a usenet post, as true now as then and just as true as it was in the 1930's when he wrote it (in at least that form).
This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.
The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.
https://xcancel.com/bradlightcap/status/2087211567012032862
It's hard to read anything into it.
It's OK though, they don't need an ethics team because everyone at OpenAI is ethical:
> “AI ethics doesn’t live with one owner or team at OpenAI and ethical considerations are deeply embedded into the model-building process driven by a number of teams across research."
But she was Meta before so can’t be a question of her principles, so indicates a personal reason
She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.
It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.
This is a very optimistic take. Most people don’t take this route, so without evidence I don’t think it should be a first guess.
No information does not mean “everything was good”. Usually the opposite.
It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.
This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.
I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.
Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.
Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.
As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.
...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.
That's an interesting data point.
He did an interview on Search Engine that I thought was interesting,[1] but it definitely seemed like there was more to the story than just, "Facebook needed to be ethical, and this guy was so good at ethics that he decided to help out."
[0] https://chochmat.org/rabbi-zvika-krieger/
[1] https://www.searchengine.show/what-does-it-feel-like-to-beli...
Even in the target-rich environment of late-stage kleptocratic surveillance capitalism, it is hard to pick out a less ethical firm which has done more moral and direct harm, to more people, with as durable a record of defying all attempts and internal and external at harm reduction.
Never use their products. Never partner. Never participate.
Oracle at least pretends to care about ethics, and already has the position.
https://theonion.com/palantir-acquires-pentagon-for-800-bill...
It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.
So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?
Ethics is also a containment vessel internally.
With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.
Head Of Ethics comes in, looks at what is really going on behind the curtains...
Head Of Ethics scrams out of there as fast as they can.
What’s next? Head of ethics for United Healthcare?
https://www.google.com/search?client=firefox-b-d&q=gavin+bel...
For that kind of money, my AI paymaster could do no wrong.
Imagine going to work and getting paid $2300+ USD per day to sit on your ass all day thinking about ethics without having to produce any deliverables. This would be a dream job for me.
If anyone questioned my morals, I'd be like; "They're paying me $550k per year" and the other person would be like "Ahhh ok, my bad! Your logic is starting to make sense now!"
Once these things are taken care of, there are so many opportunities to make so much money. If you're in the US, look at what the current US politicians and the administration is doing, and basically copy them. Right now in the US is the era for shameless greedy people, and it is extremely easy to make money from the gullible population. But again, requires the whole "shameless" part.
a) champaign socialist
b) head of ethics at OpenAI and not really listened to because you're really probably (a)
I feel like this is a complaint heard mostly in Decatur, IL.
the only ethics that actually matter come from legal.
> She has held a variety of academic positions at Temple University, Princeton, and the University of Pennsylvania – where she completed her PhD in Political Science and Government. Her Dissertation was titled “Small Talk: The Socialities of Speech in Liberal Democratic Life.”
This doesn't even feel relevant to ethics. It's adjacent, but like... I guess I'd expect a moral philosophy degree? Maybe even mathematics in there?
I find the position odd. Curious to hear what these people do and how you choose who to hire.
An ethics department is like a human resources department, but for broader ethics related matters.
HR: Ostensibly exists to support the employees, but really exists to shield the company from both a PR and legal liability standpoint when the company has a rift with one or more employees.
Ethics Department: Ostensibly exists to make sure the company acts ethically, but really exists to shield the company from both a PR and legal liability standpoint when the company acts unethically.
It is a good personal signal that you have never worked for a company with an ethics department or an ethics employee. The less ethical your company the more likely it is to have an ethics department.
Someone challenge me.
https://en.wikipedia.org/wiki/Controversies_of_Nestl%C3%A9
However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.
It’s easy to theorize ad infinitum otherwise
Or are you telling us in the hope someone makes you such an offer?
"Although the torment nexus may torment us in the short term, over the long term there will be fewer people to torment as we improve the capitalization of human's torment so that fewer people will be available to torment duing to the reduction in overall torment-forces globally, and thus, with less people being tormented, surely that will increase the over all torment reduction of the nexus. Win-win: less tormented people means less torment!
Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.
At typical corporation, something like a Chief Ethics and Compliance Officer (CECO) would be largely focused on legal compliance - making sure company personnel aren't engaged in bribery, fraud, insider trading etc. The value of this role (and the medical example) is to prevent lawsuits and/or prosecution.
For a company like Meta, an ethicist is likely concerned with the inherent ethical concerns of running a social media company, which impacts the mental health of its users, many of whom are minors. For AI companies, the ethics of releasing models that are capable of doing dangerous things. In name, the job probably involves giving recommendations about how to ethically implement features in the platform to minimize harm. In practice, the job probably accomplishes very little.
I think a lot of it depends on the size of the company. Startup? Sure, maybe it's a real thing.
Gigantic company like a member of FAANG or MANGOS? absolutely a PR move.
I can't disagree with that, reminds me of the quote from the movie Rounders: If you've been at the table for 30 minutes and you don't know who the sucker is, you ARE the sucker.
When someone isn't competent to make decisions for themselves it's ok to pressure them into engaging in behavior that will lead them to get to a condition where they can start to make decisions for themselves. In Silicon Valley they have let their guilt be leveraged by a corrupt government system that lets grifters steal billions of dollars per year in 'homelessness funding'. Those same grifters fund political groups and bribe politicians. Anyone that questions it is accused of wanting to murder homeless people.
There are many, many places in the world that are much poorer and don't have homeless encampments all over the place.
I'm not as positive as you are about what makes California (and Seattle) create/attract so many, but it's definitely an outlier compared to "everywhere".
The weather is pretty nice, much more stable than in the east coast.
Like, do you know many addicts or seriously mentally ill? Have you tried pressuring them into being better? How did it work out?
What you can do when it already happened is kill them or make them die in some ways, jail them or try to support and/or treat them.
I hope it goes without saying that letting them just die or killing or jailing them is not an option in a western society.
Prevention is much more favorable of course. For example, don't let people become homeless easily. But even then, a significant amount will become homeless.
On the street they have access to almost endless support, at least in Silicon Valley, but much of that support is contingent on things like not openly using heroin or not repeatedly getting so intoxicated that they defecate or vomit on themselves, so they refuse it.
I don't care how liberal you are, no society is functioning when it allows people to get on BART and pee on themselves and defecate on the cloth seats and walk up and down the aisles dripping feces and urine on the floor. That is allowed and nothing is done about it for fear of upsetting homeless 'rights' groups who are funded by the tax dollars meant to help homeless people.
1. Widen the definition of 'homeless' to cover people that nobody would consider homeless in every day language. Include anyone who is staying with a friend, living in corporate housing, staying with a family member, etc. Include anyone you possibly can that is unlikely to be a drug addict.
2. Narrow the definition of homeless. Don't consider people that almost everyone does consider homeless. Make sure to target the subgroups of homeless people that are very likely to use drugs.
3. Narrow the definition of drug use. Make it as hard as possible to count someone as having used drugs. Raise the evidentiary bar so high that even when you have evidence they are a drug addict it doesn't qualify them as a "drug addict" by the studies definition. Bury that definition so it's technically there but you have to go through the study carefully to find it.
Lets pick a study in a big important journal and check it for the above:
How about CASPEH: "Illicit Substance Use and Treatment Access Among Adults Experiencing Homelessness," JAMA, 2025
JAMA is a very prestigious journal:
Who counts as a substance user: Only meth, nonprescription opioids, cocaine/crack count as addictions. Cannabis, alcohol, benzos, hallucinogens, inhalants are excluded.
Someone who has a valid prescription to opiates, even if they are being abused, cannot be methodologically counted as a drug user. THIS INCLUDES METHADONE.
What counts as addict: "Regular" = 3+ times per week. A twice-weekly meth user is 'occasional use'. They might take a gram and stay up for 3 days twice a week, but they are 'occasional user' and don't count in the 37% addict number quoted by this paper.
What counts as addict: Last 6 months only. If you admit you used fentanyl every day for the last 10 years but claim you got clean 181 days ago, you are not a drug user according to this study. If they have been diverted from heroin onto methadone 181 days ago you do not count.
How did they check? They asked each person face to face whether they used drugs and which ones and how often and just took their word for it.
The real numbers from the raw data the study used:
41.6% of homeless self reported 2x per week or more hard drug use before they became homeless. an additional 23.2% claimed they started regular use after becoming homeless.
That's about 65% already from their own data, 2x what the headline pretends.
Excluding high use subgroups: to be included in the data, the homeless person had to prove they were giving informed consent. Then they additionally had to be willing and able to spend 45 minutes or so filling out the survey. The people with psychosis or who were intoxicated at the time of the interview were excluded. How many people have you walked past and they were incoherent, unconscious, or high? They were ignored in this study.
Homelessness is created by high rents. High rents are created by high local productivity. The people who are productive enough to make it over the rent-bar are housed, and their marginal productivity above the rent-bar serves to raise the rent-bar (talk to any landlord about how they set rent prices: it's the local wages, stupid). This then pushes a greater proportion of people below the rent-bar, and then they are homeless.
There are few places in the world more productive per square foot than Silicon Valley, ergo one must be extremely productive in order to remain housed there, ergo it has an extremely bad homelessness problem.
That's why people making $200k+ (an extremely productive person by national standards) need public housing assistance in Silicon Valley (https://www.reuters.com/article/world/palo-alto-considers-su...). Which of course actually just... raises the rent again...
The charities may be full of grift and whatnot, but you can remove them entirely and you will not solve the fundamental problem.
Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.
When you combine that with spiking wealth in Silicon Valley, you’ve got the perfect storm in Silicon Valley. With OpenAI and Anthropic both based in SF, it’s going to be worse than ever once they IPO.
It was actually the post-war housing boom that was abnormal, fueled by aggressive federal funding of suburban buildout. US municipalities are now broadly doomed to bankruptcy trying to maintain sprawling infrastructure that was never cost-effective to create in the first place.
Look at Austin TX. Influx of new building, prices stabilized, and now construction is dropping off a cliff. No one can afford to build into a stable or potentially falling local market. Construction will start again when local productivity grows enough for people to bid the prices back up again.
Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents.
Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents.
Go talk to a landlord about how they site prices.
Step 1. Look up income of your target demographic
Step 2. Eat 30% of their income
Can you show me a place with high local productivity and low rent?
Can you show me a place with high rent and low local productivity (that's not e.g. a vacation destination for people with high productivity elsewhere)?
You can certainly artificially suppress prices with rent control. But that just gets fed back into the system in the form of reduced supply, so it ends up raising rents on the non-controlled housing stock anyway.
Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?
Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.
No, you actually cannot do this in any lasting way.
The price of land would fall to the level required to make it construction viable again. Or the city would be abandoned, which is not generally the outcome since cities have so much shared capital investment that cannot be moved elsewhere.
Put the following prompt into your favourite AI and enjoy the reading:
"How do rents relate to productivity in the area? How does Vienna compare to San Francisco?"
Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply"
In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent.
In other words, the amount of subsidy + rent required will grow with local productivity.
Yes, it can be done, and if we want to have a reasonable future, it should be done.
Vienna's solution is fine but has escalating costs and isn't necessary. Land Value Tax is better.
Manipulating the real estate market via direct economic manipulation, collusion, or regulatory capture is so common that the entire concept of manipulating the market to earn higher profits is called https://en.wikipedia.org/wiki/Rent-seeking.
It's not just that it can, it's that it does so often it's taught in Econ 101. That you don't know this implies you have never taken an economics class or that you forgot what you were taught, and in the meantime certainly have not spent any real effort in learning about economic theory. I think you read a lot of Ayn Rand and right wing libertarian popular books about the free market and how it's perfect. Stop posting confident nonsense.
The claim is not that rents are always at their optimal or most efficient level - they're not! The claim is that you cannot (at meaningful scale) charge people who earn $1,000/month in income $2,000/month in rent to live in a place. They will have to move.
Did you take any arithmetic classes?
And any case, you should stop mind-reading. You're bad at it. I'm not a Randian nor a right-winger nor a libertarian nor free-market idealist.
Your mistake was thinking that I, the corrupt landlord who pays bribes, cares about a macro analysis of every all the properties, or that some analysis about the current distribution of incomes and possible rents will stop me from doing exactly what you say is impossible. I don't own every property in the city, especially the ones that aren't built because I capture zoning, so arguments about average GDP and average rent and whatever else do not stop me. Your analysis is useless.
ie: blaming her for anything instead of the decision makers is... really weird?
now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.
I’d argue a lot of them expect the opposite. To suppress the ethical concerns.
Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.
Do you believe that no one in management in these fields pushes the importance of ethics?
Food-testing: There's literally a parasite going around right now. Look at the response from some of the for profit businesses that was allowed to happen.
Hospitals: It is not ethical to deny patients care based on their insurance.
Clergy: The cover up of child sexual abuse scandals.
Childcare: lots of examples of sub-par care given to save money.
Her typical day: "No Sam, you can't just publicly screw over everyone and get rid of all the jobs, the peasants care about that sort of thing and if they get angry enough we'll have real problems. No we can't just kill them all, why would you suggest that?"
Not a single link to any of those. That's just rude.
There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.
But that's just one extreme example. States based on Marxism-Leninism and Maoism in the 20th century often required things like internal passports to travel within a republic's borders. If the state didn't want you visiting family in another part of the country, you weren't visiting family. That still applies to some extent in a few modern single-party socialist states.
My larger point is, a lack of ethics consumes most economic systems, because you can't easily gauge ethics like you can quarterly profits or the output increase brought about by the central committee's five-year plan. Humans tend toward math over empathy in such situations.
The capitalistic world state system was only completed in the last 50 or so years and was of course seconded by the Stalin machine. Its purpose is in the first instance to keep working populations in place, which is why all the states agree to it as they form. The system holds solid where capital moves (as did 'communism' and Comecon) but the peoples don't.
It is only when you think of individual states and not the system that you can fail to notice that we get the same result if they all try to keep their citizens in, and if they all try to keep the others' citizens out.
All of that said, no idea what this person was doing. Probably involved a red stapler.
It's performative. She got paid more to shill for a different massive amoral mega-corporation. Don't be stupid.
I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:
- A political Science major lecturing on ethics.
- Meta and ethics in the same paragraph.
- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.
- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.
Is it possible to roll one's eyes hard enough to injure oneself?
So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.
But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
By contrast, how many conversations have you personally had with people working at a company believing a profitable action was firmly unethical, but felt forced to behave that way due to "business realities" or some other whitewashing?
On the second point: my experience is rather that there is quite a variance in what is considered moral vs immortal and that often drives different decisions (in the same place, some do certain things, other don't and certain experise/people might not be available for X therefore). I cannot say that I have come across many situations where people did something they considered firmly immoral but felt they needed to go along with it (trying to recall some). Yes, finding justifications happens, too, but that isn't necessarily whitewashing as people do disagree to surprising levels.
(Changed to moral vs immoral, because that was the starting point.)
It’s much easier to disregard the human cost of decisions when you are separated from said humans by thousands of miles & billions of dollars.
History shows this is not the case.
I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.
The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.
Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.
> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
They said companies are amoral at best.
When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.
I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.
100% of the time, ethics department attracts activists, which is 100% trouble for the company in the future.
2) not impossible. imperfect maybe, but if I ask you should you buy a plane ticket or kidnap the pilot's wife and demand a free ride, which do you think gets a higher score?
3) Again, with all this Goodhart's nonsense. Goodhart's is for a minimum threshold value that is acceptable that everything degrades to, yes I get how it works and what it looks like. Throwing your hands up in the air and acting as if all is lost because some things are challenging to measure is not correct. We are not looking for things that are "barely passing the ethics evaluation" as Goodhart's "law" is focused around, rather, we are looking for things that have very high ethics scores AND completed the task well. Not just things that are "barely passing" for ethics scores. Bottom 80% don't make the cut at all - don't even think consider them as viable paths, and the top 20% we can rank according to varying criteria. Like that. It has very little to do with Goodhart's "everything approaches the minimum acceptable threshold" "law"
That gives you a three-layer picture:
Task objective: Did it accomplish what we asked?
Acceptability constraint: Did it avoid unacceptable ways of accomplishing it?
Adversarial evaluation: Can we find trajectories where the model gets a high score while violating the intended constraint?
I think what you are pointing to with your reference to Goodhart's "Law" (which is from monetary-policy and school-exams, i.e. "teaching to the test") is that the models would eventually do the minimum amount of ethics required to have an action stay valid. However, if a model is rated on ethics and it achieves the short-term-objective, then the higher ethics scoring trajectory should win. In short, 1) this is leagues ahead of where we are now for AI safety and breaking-out-of-the-lab, and 2) in baking ethics into a measurement we are adding "the spirit of the exercise" back into the maths, which is something Goodhart's Law does not account for.
And that's not the sum total of ways that the measure can fail... that's a unique way that comes into being because of the intelligence of the LLMs and other future AIs. All the normal ones are in play too, and perhaps other unique ones as well.
"Gaming" even adds a bit of an adversarialness to the process that isn't necessarily present. Plenty of measures end up "gamed" through perfectly natural attempts to maximize the measure. Someone can be perfectly honestly optimizing for "conversion rate" and not notice that they raised it by lowering the initiation rate more than they lowered the conclusion rate. "But I could account for that by measuring..." would miss the point. There is always a divergence, it only gets more subtle.
This of course also is rather glossing over the difficulty of even defining "ethical" to begin with. Some of what Silicon Valley goes to great efforts to train into their models I consider deeply unethical. Who is right? That isn't going to be answered with "whoever is the most ethical", not even in principle.
Goodhart's "law" is something that emerges when you have a constraint that says "things must be at least this tall" and gradually all things in that domain degrade to be just over that specified height. Yeah, I get the premise. The point here is that we're not concerned with meeting a bare minimum. We're outright rejecting things that do not meet a threshold, and we are also looking for a maximum. The most ethical outcome should be accepted, or among the accepted ones, that are ranked by our blurry yet better-than-nothing measurement of what is ethical.
An issue arises when the metric isn’t a good proxy for the property being measured. But ethics would not be a simple numeric target. The comment above talked about a “score”, but the question is what goes into that score. It would need to be a list of items related to topics like discrimination, advocacy of inequality, tendency to circumvent regulations, etc. Set it up correctly, and even a CEO who’s willing to “fake it” would end up being better than most major company CEOs in e.g. finance, tech, or healthcare today.