Perspective from the trenches: I teach at a university that uses Canvas. We are in our final exams period right now.
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
JumpCrisscross 7 minutes ago [-]
> the students themselves don't have the artifacts to resubmit via email because they were done in Canvas
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
e28eta 1 minutes ago [-]
Students having records of what their score was doesn't prove to the professor / university what score they received. "FWD: Exam 1 Results" is not especially auditable.
flatline 9 minutes ago [-]
Screenshot from the students’ perspective —- the group is perfectly clear, even if the administration is not.
Backups are definitely helpful in ransomwares, but before systems can be restored and brought back online, victim organizations still need to assess the scope of the breach, find the initial access vector, identify compromised accounts, and evict the threat actor. That can take time.
Gabriel54 20 minutes ago [-]
I'm surprised how few comments there are on this thread. This is probably affecting millions of students at the most stressful time of the year.
Incidentally I've always hated Canvas and probably every other LMS provider, but what is particularly amusing about this current outage is that it is occurring at exactly the time when universities are demanding that all professors put all of their materials on Canvas, without exception, due to ADA compliance regulations. It is explicitly forbidden for professors to, e.g., refer to pdfs posted on a personal website.
Other commentators here seem not to understand that many faculty also do not enjoy being forced to use Canvas.
Loughla 28 seconds ago [-]
Are you saying that making sure your courses are fully accessible to your students by following disability regulations is a bad thing?
gchallen 9 minutes ago [-]
They have not succeeded in forcing me, yet. But it's sad how many computing faculty apparently can't operate the basic online infrastructure needed to support their courses. Not that universities make it easy for us.
And of course the other serious concern I have with Canvas is that they are likely using all the materials faculty upload to train their AI replacements. Many of my colleagues engage in dark humor about this but I haven't noticed much action.
FloorEgg 2 minutes ago [-]
I'm sure the engineers at instructure are not capable of building systems that can do that. You give them too much credit.
altairprime 3 minutes ago [-]
[delayed]
dang 18 minutes ago [-]
(Comments were split across multiple threads and we've since merged them.)
Gabriel54 5 minutes ago [-]
Definitely not a criticism of your (hard) work here. Thank you!
BooneJS 1 hours ago [-]
My kids are in the middle of their finals week. What a mess. Universities know nothing, Canvas claims to be in a "scheduled maintenance", and one Prof claims to "not have any copies of material offline" which seems pretty negligent. Sounds like one section of a popular class will be doing paper exams while other sections had Canvas-based "half points for 2nd attempt"-type exams earlier today. How soon before names & grades appear in data dumps?
This would be like TurboTax "scheduling maintenance" on April 14th in the US.
corvad 1 hours ago [-]
The "Scheduled Maintenance" is just total B.S. and just honestly makes them look worse. Apparently according to their status pages this is what 99.996% uptime looks like. Pay attention lol.
HDBaseT 1 hours ago [-]
It has been over 5 hours now and there has not been any communication about this being an attack, despite many of us seeing the ShinyHunters message on the login page.
There is a lot of people who likely are unaware the latest outage is because they were compromised again.
Them marking the incident as 'Under Maintenance' means the status page isn't reporting this as an outage and adding to downtime%.
Compromised again? This is a separate in ident to the one seen yesterday?
anigbrowl 13 minutes ago [-]
Once again, an example of why corporations should not have free speech. Corporate statements that are transparent lies should be criminally actionable.
myrandomcomment 2 hours ago [-]
1. It should be illegal for any company to pay ransomware attacks. Period. No pay out ever.
2. The penalty for being the attacker should be linked to the system they violated. If you do this to a hospital and someone dies you are life in prison / chair. The minimum sentence should be so painful that it deters the attack.
No this will not stop this and companies need to be held accountable for their lack of security investment. Every attack should be investigate if the company met an agreed industry standards best practices and staffing, etc. The penalties for not meeting the requirements should be punitive.
protocolture 24 seconds ago [-]
1. It should be illegal to run insecure services. Massive Fines.
2. The payout to the hackers should form part, but not all of the penalties. Pay those guys for their great service to humanity they earned it.
parliament32 2 hours ago [-]
> It should be illegal
It should be illegal to host insecure services, especially when you're dealing with PII. Breaches keep happening and nobody gives a fuck, because the worst that'll happen is you might lose a handful of customers and buy some "credit monitoring".
Incidents like this should be followed by an audit and charges being laid. Send corp officers to jail for negligent security failures. If you can go to jail for accounting fraud, you should be able to go to jail for cybersecurity-promises-fraud.
They claim to be compliant with a number of security standards [1]. I would love to see a postmortem audit of how much of this they actually implemented.
I don't think that criminal negligence is the most helpful legal tool for incentivizing improved security. It's too hard to prove negligence.
Instead, there should be standard civil penalties for leaking various degrees of PII paid as restitution to the affected individual. Importantly, this must be applied REGARDLESS of "certification" or whether any security practices were "incorrect" or "insufficient". Even if there's a zero-day exploit and you did everything right, you pay. That's the cost of storing people's secrets.
This would make operating services whose whole "thing" is storing a bunch of information about individuals (like Canvas) much more expensive. Good! It's far to cheap to stockpile a ticking time bomb of private info and then walk away paying no damages just because you complied with some out-of-date list of rules or got the stamp of approval from a certification org that's incentivized to give out stamps of approval.
phainopepla2 2 hours ago [-]
How could you possibly make it illegal to host insecure services? Is any service 100% secure? And if it were how would we know?
I do agree with the audit and punishments for clear failure to adhere to established standards.
bawolff 2 hours ago [-]
This is a solved problem in pretty much every other domain of life - if you are following best practises but something that wasn't reasonably forseeable happens, then you're fine, but if the bad thing happens as a result of negligence then you are in trouble.
jameshart 11 minutes ago [-]
Criminal law isn't about making things alright for the victim. That's what insurance is for.
Even if you leave your door unlocked, if someone walks in and steals your stuff, it's a crime. The state has an interest in prosecuting crimes even if the victim didn't do everything they could to prevent it.
isityettime 27 minutes ago [-]
"Best practice" in cybersecurity is largely vendor-driven with little to no independent empirical validation.
That standard is likely to lock people into buying some pretty bad software, but it does little to ensure that they're running reasonably secure systems.
MagicMoonlight 7 minutes ago [-]
In civil law maybe, but you aren’t allowed to blame a rape victim for choosing to walk down rape alley…
hsbauauvhabzb 2 hours ago [-]
No building has a 100% chance of not caving in, yet somehow I think charges would be laid if a skyscraper caved in.
jameshart 7 minutes ago [-]
This analogy seems to be portraying 'ransomware hackers' as an unstoppable force of nature akin to gravity.
I'm not sure that's a fair analogy.
primitivesuave 40 minutes ago [-]
If Boeing claimed a plane was airworthy, but it crashed because basic engineering controls were skipped, we have collectively put our faith in the NTSB to preserve evidence, run an independent technical investigation, etc. There is no such authority for software - most security auditors (SOC2, HITRUST, etc) are just looking at self-reported data.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
a34729t 1 hours ago [-]
Has a corporate officer ever gone to jail or been meaningfully fined for a data breach?
mikeweiss 2 hours ago [-]
Shouldn’t we be focusing on making it harder to pay overseas criminals in the first place? /ahem/ crypto platforms facilitating transfers to bad actors /ahem/
Bud 11 minutes ago [-]
[dead]
bombcar 2 hours ago [-]
Your "minimum sentence so painful" will certainly dissuade foreign nationals, even foreign governments.
Kostchei 2 hours ago [-]
interestingly, having actually done the law enforcement side of these investigations, 50% of them are local. And I understand that this is not 100% solution, but neither is any form of law enforcement, but that doesn't mean we should fail to attempt it.
Kids from the local uni having a lark, stalkers, vindictive ex employees, local gangs, criminals who understand their victims because they hail from the same community. These are your local hackers. Sift them from the nation states and international crime groups, then deal with the International as a matter of diplomacy. Because we do this so poorly locally, we have little ammunition to when it comes to diplomacy. "reduce attacks by your crime groups and we buy your natural gas, seel you wheat etc"
Want more motivation?- 75% of the local attacks by volume send funds back to terrorist or separatist organizations.
It is not an in-soluble problem. Sentences are a fraction of the answer, effective and receptive reporting processes are more important, then government backing for investigation and enforcement, then policy around home-team activities (ie don't do the bad things yourselves Mr Gov). Deterrence comes after all that.
Aurornis 1 hours ago [-]
One tech ransom case I know of was an inside job. It definitely happens.
There are already significant penalties for doing anything like this. The guy involved is in prison for a very long time. I don’t recall the exact number of years but I do remember it was so long that he wasn’t going to see his kids grow up.
I don’t think anyone who puts a little thought into a crime like this doesn’t understand that the penalties are already very huge. You don’t get a slap on the wrist for extorting a company (or person, for that matter)
hluska 1 hours ago [-]
50% of ransomware attacks are local to where? You’ll need to cite some sources because I don’t believe that is possible.
nullsanity 45 minutes ago [-]
[dead]
da_chicken 2 hours ago [-]
Yeah, they identified themselves as ShinyHunters, and the IP they've put on the demonstration page is geocoded to Russia. Notice this is the same group responsible for the Infinite Campus hack last year.
Really, though, if you want someone to blame, Instructure is not a particularly compelling target. Let's review:
1. Iran is intentionally targeting infrastructure due to a war started by the current administration.
2. China is actively seeking corporate secrets to steal and commercialize for themselves, spurred by extreme protectionism and retaliatory tariffs.
3. North Korea is doing anything they can -- including just taking a remote job by proxy -- in order to extract any money.
4. And Russia is working with and aiding all of them, after everything else going on has forced the embargo to break.
5. All of this while completely alienating every single one of the United States' allies.
6. Meanwhile, the American DHS is currently shut down.
7. And this is after Trump cut funding and personnel for CISA severely enough they've had to end the contract with MS-ISAC, meaning all state and local entities can only remain in the organization if they foot the bill for it directly and CISA and other agencies responsible for cybersecurity are more thinly staffed than they have been in decades.
In short, the current administration systematically disassembled all the protections we have built over the last 100 years, and then placed infrastructure -- schools, in this case, but also power companies, water treatment facilities, communications companies, local governments, hospitals, food producers -- directly on the front lines of the modern geopolitical conflict.
That vast ocean that has kept us safe historically is a poor moat in the modern era.
elictronic 1 hours ago [-]
Complete internet blockage of nations allowing the attacks. If foreign governments are you can always execute them. We are living in a different world where this is no longer a zero probability occurrence.
Bud 9 minutes ago [-]
[dead]
pants2 2 hours ago [-]
When will countries start treating cyberattacks as an act of war? If the North Korean military came to America and robbed fort Knox of $200M in gold there would be retribution. But hack an American company for the same amount and the feds do nothing.
prodigycorp 2 hours ago [-]
Ok, so we treat it as an act of war. Now what? Attack North Korea? Great, the entire city of Seoul gets shelled within five minutes of your attack and hundreds of thousands of innocent people die.
It's very easy to play with lives that aren't yours.
toraway 10 seconds ago [-]
Exactly. This is the "Declare fentanyl a WMD" of solutions to ransomware. Sounds kinda badass as long as you don't overthink spend too long think about it but has no practical relevance to actual enforcement challenges.
a2128 2 hours ago [-]
How do you know which country to blame? It is standard practice for foreign actors (or just hackers in general) to use proxies around the world to misdirect and insert false clues as to their origin. It could be an American teenager proxying through North Korea, and it could be a North Korean proxying through another American teenager's residential connection, there's no way to know.
bigyabai 2 hours ago [-]
They already do. This is what asymmetric warfare looks like, your weakest links will break in a time of crisis. Focusing on retribution for the Dunder Mifflin cyberattack is pointless, the adversarial motivation is purely to disrupt and extort.
The best response to a cyberattack on critical systems is to take security seriously. Document the offense, avoid the same mistakes and invest in penetration testing. Of course, nobody is incentivized to do that until they're attacked, so the cycle perpetuates itself.
dev360 28 minutes ago [-]
> No this will not stop this and companies need to be held accountable for their lack of security investment.
I think in principle, its sound. Im also just baffled hearing anecdotes from friends that are in big corp world and hearing the type of incidents they have, and how they respond to it.. It makes me wonder if there is enough capable talent to go around for the "boring corp" crowd.
Hint: I don't think there is nearly enough talent to go round, but for these companies, its either that they think they have solid experts (and didn't), OR its not a real priority until you get hit.
charlie90 42 minutes ago [-]
If someone robs a bank and someone inside dies of a heart attack, thats felony murder. I would be happy if the same applied to ransom attacks or other blackmail/leaking of info. If someone commits suicide because of it, its murder.
Avicebron 2 hours ago [-]
We could also throw the CEOs of companies who don't properly secure their infrastructure and pay their security engineers enough in jail. A little justice on both ends.
scheme271 2 hours ago [-]
Uh, who determines that the infrastructure wasn't properly secured? Who is willing to risk prison because some intern accidentally committed an API key or made a dumb mistake. Conversely, what's the chances that no one actually gets prosecuted regardless of how sloppy their security practices are?
applfanboysbgon 2 hours ago [-]
> who determines that the infrastructure wasn't properly secured
An investigative body, the same kind that determines the who, the why, and the how when an airliner crashes or a bridge collapses. Obviously a lot of work needs to be done to get from point A to point B, and it won't happen overnight, but software development is currently a deeply unserious profession and at some point a genuine software engineering practice needs to be developed.
I am, perhaps naively, slightly hopeful that the LLM bullshit plaguing our industry will be the gust of wind needed for the house of cards to collapse and governments to realise that allowing the entire world to be vibe coded is not sustainable.
dghlsakjg 1 hours ago [-]
Pretty famously, aviation incident investigations are almost always not done with prosecutorial intent, and more about truth finding. It leads to people involved being cooperative to prevent future problems instead of ass covering to prevent jail.
Aviation’s safety record is not coincidental.
Avicebron 2 hours ago [-]
Ideally the chances are high to certain they get prosecuted for sloppy security practices. It's part of the gig of being a CEO, if you imagine you are such a visionary/ideas guy/leader/whatever, risk taker (always a risk taker) then you can gamble spending 20 to life because you weren't actually as good as you thought.
kelnos 1 hours ago [-]
A friend who teaches at MIT said they were hit by this. I found it ironic and a little sad that a place like MIT doesn't have an IT staff that can maintain their own on-prem solutions for things like this.
But it turns out that MIT used to have their own homegrown system, and recently switched to Canvas. Bet they're regretting that now.
The build vs. buy decision seems to have swung very hard toward buy in the last decade, and I think that's a shame. Yes, orgs need to focus on their core competency, and sometimes that means outsourcing things that aren't core competencies to third parties. But there are always downsides.
mingus88 33 minutes ago [-]
I started my tech career in EDU. I’m not at all surprised.
IT staff who are ambitious and talented don’t last long in education. The pay is very low compared to industry. Where I worked, you could retire with a comfortable pension after a number of service years, so the IT staff outsourced as much as possible so they needed to take zero risks to their nest egg. Blame all the problems on the consultants and do as little as possible.
It’s literally where dreams go to die.
MIT is known for the brilliant professors and students but at the end of the day, running a university is pretty standard stuff. They don’t need a genius rockstar to admin the courseware servers.
corvad 2 hours ago [-]
Canvas is handling this terrible. No communication, no status updates, etc. Also looks pretty bad their whole platform was compromised and not a single real report for the breach that already had happened. Wonder how long it will take for SLA violations and lawsuits to manifest, especially with most U.S. schooling having finals right now.
user3939382 1 hours ago [-]
Lot of experience dealing with Canvas/Instructure. Tech is o-k. Culture seems to be full of themselves due to market position.
corvad 1 hours ago [-]
Yeah like their page says "Scheduled Maintenance" which is total B.S. Talking to people at my university's IT side of things Canvas has said nothing to any clients.
bumblehean 5 minutes ago [-]
Hugs going out to the teams at Instructure working to fix this. I've been through a similar Ransomware attack (national news stories, lots of customers dead in the water, etc.), and it's about as bad a situation you can wind up in.
SoftTalker 4 hours ago [-]
So many universities used to run homegrown or on-prem student systems. This is the downside of consolidating in the cloud. If the infrastructure is compromised, it affects everyone, not just isolated or single installations. I wonder how they are feeling about that decision now? I guess they can say "not our fault" so they might be feeling better than if it was a vulnerability in their own system.
crazygringo 3 hours ago [-]
If an exploit is found in the software, hackers will often be able to attack hundreds of separate institutional installations in an automated way just as easily. And depending on the exploit, potentially more easily if on-prem admins fail to take all recommended security steps.
I'm actually much more interested if there is any financial liability for Instructure here? It's interesting that it's the universities being ransomed, while the technical failure was Instructure's. We're used to uptime SLA's -- what about security breach SLA's?
harikb 3 hours ago [-]
> It's interesting that it's the universities being ransomed, while the technical failure was Instructure's.
My guess would be they get likelihood of getting paid when blackmailing 9,000 schools (at least a few would pay up) than blackmailing Canvas/Instructure.
I don't think any SLA/terms would change who gets to feel the pain.
poopmonster 3 hours ago [-]
My guess is that they believe by maximizing their attack coverage, the odds are greatest that some of the institutions will pay up. And otherwise, they can still make a bit of money by selling the data.
Don't ransom all your eggs in one basket
dylan604 2 hours ago [-]
Yeah, if they had spent the time and money to roll their own that got hacked, they'd be responsible. Now, they can just clap their hands and show them palms up to you like a black jack dealer and walk away from the table with no responsibility. Probably one of the biggest benefits of using a product instead of building your own.
kelnos 1 hours ago [-]
It's annoying that this is how internal politics usually works. Decision-makers at an org should be considered just as responsible when a third-party choice goes bad as when an internal tool goes bad.
frollogaston 2 hours ago [-]
It's still more secure this way, especially with AI hacking making it harder to rely on obscurity.
Also yeah there is value in being able to blame another party, and also being down when everyone else is down.
rahidz 3 hours ago [-]
Goddammit. Anyone in the know, know if Parchment was also impacted by this potentially? They were acquired by Instructure a few years ago, and deal with a LOT of transcripts.
Edit: https://status.parchment.com/ says "While Canvas, Canvas Beta and Canvas test are currently unavailable, we are simultaneously monitoring all of our other product environments, including Parchment. We continue to see no reason to believe any Parchment resources have been impacted."
matthewfcarlson 3 hours ago [-]
I remember circa 2010 a friend of mine at college was like “blackboard sucks, let’s build something new”. At the time I poo pood the idea and lo and behold canvas came out a year later. Outside looking in, they been crushing it.
HPMOR 3 hours ago [-]
One of my mentors created Blackboard. It used to be very very good, but he sold it to private equity, and they immediately fired all of the customer support and developers, 3xd prices overnight leading to the 'blackboard sucks' problem. This gave the opening for Canvas to eventually come on to the scene and dominate.
My wife and I each have to use it as we're both following an online master's at the same university... it's definitely gone downhill (compared to the days where I originally used it ~20 yrs ago in college; tracker-riddled, slow); surprisingly, a recent change made it so that you can only attend online lessons in Chrome (haven't had time to see if this is just a user-agent thing).
I worked in a college IT department around that time and the common belief was that all LMSes suck. There are just too many different ways that too many different people want to do things that it's just bound to be hated. Kind of like Jira / Asana for software dev project management.
SamuelAdams 54 minutes ago [-]
LMS’s are a lot like programming languages. There’s the ones people complain about and the ones no one uses.
asdff 2 hours ago [-]
I used both and could not tell you the major differences. I feel like they are equivalent in the bread and butter features. Most people don't use 99% of the functions they bake into these. Just use it to hold the syllabus, maybe hold the slides, submit assignments, and spreadsheet for grades. All stuff you can do with email + spreadsheet already. Maybe throw in a shared drive for larger files, which every university in the country already pays for.
quadrature 2 hours ago [-]
"Equivocal describes something ambiguous, uncertain, or open to multiple interpretations, often used to intentionally mislead or evade."
do you mean equivalent ?.
asdff 2 hours ago [-]
yes
vlunkr 1 hours ago [-]
Blackboard got a lot better in response to the flood of customers heading to canvas.
kayyyy 2 hours ago [-]
As someone who has used both as a student and a TA I find blackboard miles better, much easier to find what i'm looking for and my professors seem to have better luck laying out their course on blackboard than canvas.
breakingstuff 1 hours ago [-]
I actually disagree, based on my time using Blackboard as an admin, student, and teacher. Although my experience is a few years out of date, I found the interface cumbersome and the performance slow.
smurda 2 hours ago [-]
Blackboard, the Canvas predecessor, was so unstable that we called it BlackOutBoard
forgetfreeman 2 hours ago [-]
They are definitely crushing it on sales. The actual product is a radioactive dumpster fire that is simultaneously hostile to students, teachers, and parents.
dghlsakjg 1 hours ago [-]
Yeah but the customer is the administrators who never have to make contact with the real world
thecatapps 3 hours ago [-]
I remember when I was in high school (2016? 2017?), I found a super simple XSS in the assignment submission form and told the programming teacher. Canvas then proceeded to lock my account and got me my first (only?) detention. Good times.
somebudyelse 7 minutes ago [-]
Somewhat similar vein, the school's blocking software would block YouTube and embeds unless they came from Canvas. They were smart enough to disable the HTML editor for posting discussion comments, but forgot that since it was a rich text editor, you could just copy-paste in an embed by putting the code in data:text/html, then copying the element as formatted html.
I also ran the entire DOMPurify sample XSS and managed to find one way to download custom content onto someone's computer.
frollogaston 2 hours ago [-]
Uh, did you tell the teacher by exploiting the vuln?
exprez135 6 hours ago [-]
The Canvas instance at the nearby university is now down (May 7, 4 PM Eastern), but was briefly displaying the message in this screenshot (1). The ransom message implies that today's problem is the second wave in an attack on Instructure after ignoring their first breach in recent days.
We received communication that Canvas is down for "Under Maintenance" although it seems ShineyHunters have compromised Canvas again with that message you posted.
Seems like Canvas instances of schools not listed are also down (at least my alma mater is)
goldenskye 3 hours ago [-]
Yes, I work for an Australian online school. We’re down “for scheduled maintenance” (I question how “scheduled” it was given this is within school hours on a school day), but we’re not on the list published by ShinyHunters.
avs733 2 hours ago [-]
our instance went from [insert hacker leet text] to "down for scheduled maintenance" and myself and other faculty are just having the darkest humor about this.
Some instances seem to be recovering. I wonder if a ransom was paid.
somebudyelse 6 minutes ago [-]
It looks like Instructure has been removed from the ShinyHunters website. Both the entry and the list of schools has been removed.
OsrsNeedsf2P 2 hours ago [-]
Somehow I have less distaste for ShinyHunters than I do for the companies who don't secure user data
tptacek 48 minutes ago [-]
The boy is a biochem PhD student at UIUC and reports that all their finals are now cancelled. "Is this good news?" I ask. "Yes. Everything coming up Milhouse."
orourke 50 minutes ago [-]
My son was in the middle of an exam and then his screen went black and it showed the message from ShinyHunters. Hasn’t been able to get back in since.
somebudyelse 2 hours ago [-]
It looks like Instructure has been removed from the ShinyHunters website. Both the entry and the list of schools has been removed.
bombcar 2 hours ago [-]
Look for large BTC moves recently?
corvad 2 hours ago [-]
Ransom paid?
robertritz 1 hours ago [-]
I'm shocked universities don't host their own LMS? At least large universities have the IT departments to do this. They host compute clusters, so they can certainly host an LMS.
oezi 25 minutes ago [-]
The same reason hospitals don't have their own Patient Information System but all use Epic. The amount of customization you need and continuous churn due to changing curricula and regulatory requirements makes it hard to keep up without scale.
owlboy 26 minutes ago [-]
I’m not surprised. Canvas kind of sucks. And their development is slow. And they are poor at communicating during mundane events.
stringfood 17 minutes ago [-]
They're also apparently poor at communication during highly interesting events as well
tom1337 5 hours ago [-]
> Canvas is currently undergoing scheduled maintenance
doesn't seem that scheduled to me
javawizard 3 hours ago [-]
ex-Instructure employee here (though it's been about 10 years since I worked for them).
That's just the quickest page/status update to throw up; it was a one-liner to push it live back when I was on the deploy rotation.
I'd hazard a guess they have more important things to worry about right now than exact status page messaging ;)
anematode 4 hours ago [-]
Well, scheduled by whom? :)
3 hours ago [-]
mystraline 4 hours ago [-]
Whoever it is, is likely defended by Cloudflare. They seem to like the booters.
Just learned the defacement page was hosted from instructure's own aws bucket so seems pretty bad.
sharkweek 3 hours ago [-]
My wife is in grad school at a major university and is dealing with this right now the week of midterms for spring quarter.
I totally understand why a university wouldn’t want to bake their own learning portals but just feels like such a single point of risk to use third party solutions for something like this.
Back in my day… all we had was a school email via on-premise services. I guess we registered for classes in a web portal but that’s about it. The idea of online class was entirely foreign at the time. Ain’t nobody hacking a blue book.
gdhkgdhkvff 2 hours ago [-]
It’s wild to me that people in this comment section are suggesting that schools should improve their security by rolling their own platform, which is bound to be filled with security holes, instead of using a popular, maintained, open source option.
nazgul17 1 hours ago [-]
To be fair to the idea, though, while this would make individual instances less secure, it would drastically decrease the leverage for the work bad actors put in.
There is a saying in the software security industry that (I'm paraphrasing from rusty memories) a system is secure if the cost of hacking it is higher than the value it protects.
Each system being completely distinct from another means that the cost of hacking the average student goes up by 9000 (from the article, Canvas is used by 9000 schools).
Still not saying that rolling out your own is the preferred solution, but the idea is not as ludicrous as it would seem, and should definitely be entertained and discussed, at least.
forgetfreeman 1 hours ago [-]
Maybe. I still remember the Drupal community sneering at the New York Times when they unveiled their homegrown online news platform bitd. After 15 years of recursively scraping ad-hoc porn sites off of server hard drives when clients dragged their feet on migrating to latest versions I 'm less certain the assumption that homegrown == less secure is as valid as it sounds.
asdff 3 hours ago [-]
Universities used to do this sort of stuff themselves. Then it became a business handled by purchasing rather than needs met by the department themselves.
afavour 2 hours ago [-]
In fairness in the era where universities did it themselves the tech requirements and expectations were dramatically lower.
asdff 2 hours ago [-]
Tech requirements are the same as they always were. One needs to ask whether they need so many frameworks to host some files on the internet and submit some files and perform spreadsheet calculations. We still used one of those First Age 1990s websites for sort of pre lab quizzes this one class when I was going through it, and it might have looked a little "old" but I mean it did the thing and worked for years and will continue to do the thing and work for years.
internetter 1 hours ago [-]
You're being deliberately obtuse. Canvas has many many features. Wikis and discussion boards and quizzes (with some anticheat) and groups and the list goes on and on. Furthermore, while it was never the flashiest thing, it did it better than many of its predecessors. Yes, an individual class may not use all of these features, and yes canvas has suffered feature creep even over my time as a student and yes canvas is not doing anything technically challenging, but there is enough of it that each school rolling their own everything would be a drastic waste of everybody's time and money.
clipsy 2 hours ago [-]
Have these dramatically higher tech requirements and expectations improved the quality of education whatsoever?
avs733 2 hours ago [-]
Because faculty didn’t want to do it anymore. They want it handled by others but also they want oversight and veto power but also they don’t want to be bothered. But it better always work, and if they make a mistake the software is broken because don’t tell them it’s a user error they used to write Fortran.
As a faculty member at a large university…I have a deep respect for the impossible job of university IT departments.
We originally rolled our on LMS decades ago. When we switched to canvas we kept the home brew running for five years past its expiration date because faculty refused to remove their files. Finally each one was manually moved by IT for the recalcitrant old faculty.
asdff 2 hours ago [-]
It is kind of funny when these LMS tools with 100+ functions are being used for little more than what email, a grades spreadsheet, and maybe a shared drive would do. University might even ask for the final grades in spreadsheet format by the end of the term anyhow, so data goes into the LMS just to come back out again.
avs733 1 hours ago [-]
In a sense you aren’t wrong but those analogies fail at scale. It’s like saying you could replace gt functions with a spreadsheet.
They are large databases yes but they do a lot of small and large things that that analogy glosses over
jagged-chisel 3 hours ago [-]
> Ain’t nobody hacking a blue book.
Well not with that attitude
ibgeek 2 hours ago [-]
Moodle is an open-source LMS that can be self-hosted.
Another open-source LMS that can be self-hosted is... Canvas.
wmoxam 1 hours ago [-]
Almost no one does
ibgeek 2 hours ago [-]
Didn't realize that. Thanks for the info!
walrus01 1 hours ago [-]
A university doesn't need to bake its own learning portal, Moodle exists and is used by a lot of large schools.
userbinator 2 hours ago [-]
I totally understand why a university wouldn’t want to bake their own learning portals
They used to, in the pre-cloud/SaaS era; and they were much simpler and better UX than the slop that they're renting today, because the actual users were not far from the developers.
oezi 29 minutes ago [-]
Counterpoint: I was a PhD student in 2004 and on the universities board* which oversaw the roll-out of the campus management system. It cost > 10m EUR to implement a shitty system with the worst UX and years of stabilizing to make it somewhat work.
The amount of corner cases and performance requirements during rush times (semester start) made it really infeasible for a university to roll their own.
* German universities have this funny system where 51% of such boards are controlled by the professors and the rest is made up of other employees/staff and students. They call it academic participation.
incomplete 5 hours ago [-]
yep, i work for a major university and our canvas instance is down. this is really, really bad.
tbh this has me wondering if canvas "instances" are actually as isolated and segregated from each other as they're supposed to be.
javawizard 3 hours ago [-]
Define "as they're supposed to be".
Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
wky 4 hours ago [-]
It's possible that Instructure's servers got compromised:
dig canvas.ucdavis.edu
[...]
;; ANSWER SECTION:
canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com.
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15
ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18
dig canvas.duke.edu
;; ANSWER SECTION:
canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com.
duke-vanity.instructure.com. 60 IN A 18.173.121.125
duke-vanity.instructure.com. 60 IN A 18.173.121.18
duke-vanity.instructure.com. 60 IN A 18.173.121.103
duke-vanity.instructure.com. 60 IN A 18.173.121.15
mrsvanwinkle 4 hours ago [-]
that's what the screenshot says. They rooted Instructure servers.
SamuelAdams 3 hours ago [-]
It depends on what you pay for. If you need FedRamp or IL4+ compliance you are likely on dedicated infrastructure. Everyone else uses multi tenancy.
starkrights 3 hours ago [-]
The source txtfile has since either been dos'd or deleted (at least it was when I tried to access)
Someone dumped the content into a google doc on reddit[1] if anyone's interested.
A college student I know just sent me a screenshot, he can't access canvas for his school at all
yesiamyourdad 6 hours ago [-]
Same, my daughter just sent a screenshot, she was trying to study for finals.
ThrowawayR2 3 hours ago [-]
I wonder when the public is going to start calling for corporate liability for malpractice in software development and corporate liability for malpractice in IT deployments. Even if the tech industry fights it, it probably won't be that much longer.
brendanyounger 3 hours ago [-]
I'll never understand this point of view. If someone would please explain how to create perfectly secure software, I will gladly start writing perfectly secure software. Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths. Malpractice insurance is high. Litigation is constant. And patients still die on the operating table. It's unclear what all the malpractice tort law actually gets you in the end.
cortesoft 2 hours ago [-]
> Only after, if it's clear I ignored obviously correct advice, should there be malpractice penalties.
In most of these cases, the companies involved did NOT follow standard security practices.
I am pretty sure that is what people mean when they say "held responsible", they mean "held responsible for failing to follow standard security practices", not for the actual act of getting hacked.
ThrowawayR2 1 hours ago [-]
> "Consider surgery instead of software development."
Is that really the analogy you want to use the bolster your argument? Licensing was forced on the medical profession because of rampant quackery causing a large number of deaths. Some of the horrors that went on before enforced medical licensing are well-nigh unbelievable, e.g. https://en.wikipedia.org/wiki/John_R._Brinkley
2 hours ago [-]
kelnos 1 hours ago [-]
I agree that even if companies do everything right, they can still get popped. But most companies do not do everything right, and they should be legally responsible for those things.
But even if they do everything right, is it really fair to let the companies just shrug their shoulders and say "it happens"? While their users are the ones who really get hurt.
dylan604 2 hours ago [-]
> Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths.
I like this analogy, but deaths shouldn't be the leading indicator just an indicator. Family member had a surgery with well known procedures, say removing a gall bladder. Unfortunately, this surgeon skipped a step in lieu of setting a record for fastest procedure. Because steps were skipped, the gall bladder was not scooped into a net to avoid spilled gall stones which resulted stones spilling into the abdominal cavity requiring numerous follow up surgeries to remove the spilled stones as they made themselves known. So clearly not following accepted procedures should be a clear win in a malpractice case, yeah? Wrong. No doctor would testify against the surgeon and the case was dismissed. I feel like this is exactly how it would work in software security incidents as well.
dctoedt 59 minutes ago [-]
> this surgeon skipped a step
That was the foundational premise of Dr. Atul Gawande's book The Checklist Manifesto, an expansion of his article The Checklist in The New Yorker [0]
Well, you don't know how many more would have died if doctors and hospital didn't care about their insurance going higher???
cortesoft 2 hours ago [-]
I do wonder if that won't just end up INCREASING ransom-type attacks, though?
If we increase the penalties for a company being hacked, you create even MORE incentive for hackers to try to break in, because if they succeed, they have a pretty big stick to threaten companies with when demanding a random payment - not only will the company have the negative effect of the data being leaked and the PR that accompanies it, they now know that if they don't pay and the attack becomes public knowledge, they face a big fine or other punishment.
A company is much more likely to pay a big ransom if they know they are just going to end up paying that much or more in fines if they refuse the ransom and report the hack instead.
If you take this route, and increase punishment for being hacked, you are making a pretty big bet that the main reason companies are hacked is because of poor security practices. I am not sure if that is true or not.
I wonder how much old data Canvas keeps around? Are students who graduated in 2016 going to be at risk of having their academic data leaked?
Fumblenuts 10 minutes ago [-]
I bet it depends on the institution and the IT team behind said institution, but at least for my university we apparently don't delete old course shells or anything.
I'm friends with a professor who complained to me a couple times about how sometimes he will need to scroll through pages and pages of courses he taught in the past. He also mentioned that profs aren't able to delete their own course shells either.
QLD Government vendor selection is always terrible.
goryramsy 4 hours ago [-]
Down for all students at my University… it’s going to be a headache for all professors to deal with extending due assignments.
bigfatkitten 6 hours ago [-]
I use Canvas for some postgraduate studies, and my teenage daughter uses it at her high school.
We already bond over how awful the Canvas UX is (and she has a bunch of Chrome extensions to improve it.) Now we’ve got something else to gripe over together.
Canvas seems like it’s not that great. But if you then use Blackboard Ultra it makes canvas look amazing.
copperx 4 hours ago [-]
I vibecoded a pretty extensive CLI for Canvas and using it is very pleasant. Joyful, even, when combined with an LLM. Especially when compared to the developer hostile Blackboard Ultra.
flashman 3 hours ago [-]
What's in the files they've already released? Some of them are > 800GB.
HDBaseT 2 hours ago [-]
Where are you getting that information from?
I'm under the impression files are getting released 12th May.
I don't see any reporting on 800GB?
DauntingPear7 2 hours ago [-]
Grades, records, etc I would assume. Someone else pointed out that they recently acquired https://www.parchment.com/ so they may have also been able to scoop up those records too
emmelaich 2 hours ago [-]
Also discussions between students and teaching staff.
poopmonster 3 hours ago [-]
I'm guessing loads of student work? If so, it'll be great for anyone who wants to research AI usage in papers.
3 hours ago [-]
poopmonster 3 hours ago [-]
Student at an impacted university here.
Our whole testing center is down. This is inconvenient, but mainly it's amusing. I swear strangers are talking to each other more. I'm noticing people just sitting in the sun and relaxing. Nature is healing.
(Of course, plenty of people have also just finished their exams, so it's hard to know the cause.)
Any idea what data Instructure-and-also-now-ShinyHunters even purport to have beyond names, profile photos, pronouns, homework assignments, school communications, phone numbers, and email addresses?
i.e. What makes this threat so different from what any old data brokers have already scraped?
What leverage besides aura farming do the ShinyHunters really have?
All I can think of that's really valuable is passwords. And private communications in Canvas DMs. But if you're being at all intimate over your school email, that's kinda on you.
Anyway surely Instructure only stores user public keys or something?
Alternate history question: If they just sold the data, never revealed the hack, and didn't make a scene, from a customer perspective, how different would this be from business as usual?
plasma_beam 6 hours ago [-]
Our public school system here in Maryland got hit, ransom screen.
5 hours ago [-]
3 hours ago [-]
avs733 2 hours ago [-]
It is absolute chaos at my institution. This is the last day of finals and grades are due Monday morning. Most faculty are spending today, tomorrow, and through the weekend finalizing grades.
What we don't have access to includes:
* Already graded work
* Ungraded work
* overall adn assignment grades
* lists of students and student emails from the course
* messages from students that are often sent through gradescope
Just...complete implosion.
pesus 2 hours ago [-]
What happens if the system isn't back up in time for grades to be submitted? Just a delay?
eatmyshorts 4 hours ago [-]
My daughter says that Northeastern is also affected. Is it more widespread? Did they infect all SaaS Canvas universities?
vondur 4 hours ago [-]
It looks like every CSU System is on the list (California State University). Surprised this hasn't hit the front page yet.
DaSHacka 4 hours ago [-]
Possibly because they haven't released the data yet?
I'm honestly surprised more people aren't talking about this.
wg0 33 minutes ago [-]
You learn all the technical details only to harm people like that instead of making a modest and honest living.
Shame on your existence basically.
bagels 3 hours ago [-]
It's been a long time since I was in school. What does this software do?
mbreese 3 hours ago [-]
It is how classes (even in person ones) are organized. Assignments, quizzes, links to online textbooks, discussion boards, student/teacher messaging, student group messaging, etc. From the teacher side, I'm not sure if there is a backup copy for things like grades outside of Canvas. It's that pervasive.
Everything from middle school up to grad school.
It's a particularly interesting time to have this happen too -- many finals going on now.
windows_hater_7 3 hours ago [-]
It’s a “learning management system.” It replaces a course website in most instances. It’s also used for course grades and you can submit assignments or take quizzes.
If you’re a student or teacher: nearly everything that matters. Homework, materials, lectures, grades. It’s all on canvas.
podiki 5 hours ago [-]
And grades are due in the next week or so for many of these (usually a quick deadline at the end of the semester due to graduation happening)...
enjo 4 hours ago [-]
My wife’s grades are due tomorrow. She was in the middle of finishing exams when it happened. She can’t even access the exams to grade by hand. Total mess.
SoftTalker 4 hours ago [-]
Graduation is just a ceremony. The actual credential award depends on whether you finished all your coursework and is not time-boxed by that event.
Of course if you can't complete your exams because of this, that's more of an issue!
skeaker 6 hours ago [-]
Pretty cruel to do this right around finals.
kelnos 54 minutes ago [-]
That's exactly the point, I'm sure.
crazygringo 3 hours ago [-]
Even more incentive to pay up. I wonder if the timing was intentional or just coincidental.
enceladus06 3 hours ago [-]
That is the point. Get an extra million or two $ in btc from Instructure.
daledavies 6 hours ago [-]
Eek I bet there are a few people at Instructure who won't be getting much sleep tonight!
gigel82 4 hours ago [-]
Damn, all schools in our district in Washington moved to Instructure last year.
They moved away from Teams because it objectively sucked, but I haven't heard of widespread compromises like this in Microsoft's systems so...
SilverElfin 1 hours ago [-]
Terrible that this affects children and that their information may be ultimately leaked. They need to be greater consequences in the law for security breaches.
vinni2 4 hours ago [-]
I hate Canvas. I would rather run a course on GitHub. But our university forces it on us. And now this.
crazygringo 3 hours ago [-]
Do you remember how Canvas was a gigantic improvement over Blackboard?
And GitHub doesn't provide a way to record grades that remain private per student last I checked, much less sync them to the university, or 99% of other things Canvas does.
I don't love Canvas, but it's far, far preferable to a world without it.
poopmonster 3 hours ago [-]
It is really convenient and stays out of the way. As much as I'm enjoying the mess, I am forced to appreciate its value.
bombcar 2 hours ago [-]
> remain private per student last I checked
last I checked it appears grades remain private per planet or so ...
bombcar 2 hours ago [-]
How does Canvas compare to things like Moodle?
Or is it an entirely different class of beast?
wmoxam 1 hours ago [-]
I've written a bunch of LMS integrations so I've had the opportunity to use all of the major LMSs. Basically, all LMS systems are rather user unfriendly and complicated with a ton of customization options hidden under layers of sub-menus/configuration settings. At their core they provide a grade book, student management tools, and some basic CMS type functionality for posting class messages/coursework/etc. They've all adopted a standard for interacting with external tools (LTI).
Canvas generally is the 'easiest' to use, and the 'cleanest' looking one although D2L Brightspace is pretty good too. Moodle out of the box is pretty confusing and ugly, but I've seen some heavily customized instances that look a lot better. Blackboard is the worst of the bunch IMO.
frollogaston 2 hours ago [-]
Wow, I last used Moodle in 7th grade, 2008. It seemed like a similar thing.
jrm4 2 hours ago [-]
Canvas shouldn't exist in its current form, and neither should have Blackboard.
It's always been as stupid as requiring that your chalkboard, chalk, chairs, bluebooks, pens, paper, gradebook etc etc all come from the same company.
I, for one, am very much looking forward to my IT Gov council meeting tomorrow.
swatson741 1 hours ago [-]
I saw this happen to my Canvas account today. At first I thought it was a prank from the school or Instructure. The message was sent to students which makes no sense. Second, the message that was sent basically implies that ShinyHunter is actively getting patched out, and no one is ever going to give into their demands. They're basically saying that they're done and desperate. It's a strange message for ShinyHunter to send, but I think they were trying to pull off a psyop / FUD.
Looking into the payload they sent me this is how they hijacked the screen. Everything in the payload is unchanged except for one line of code:
body::after {
content:
"\A\A"
"S H I N Y H U N T E R S"
"\A"
"rooting your systems since '19 ;)"
"\A\A\A"
"ShinyHunters has breached Instructure (again)."
"\A"
"Instead of contacting us to resolve it they"
"\A"
"ignored us and did some \201Csecurity patches\201D."
"\A\A"
"\26A0 W A R N I N G"
"\A\A"
"If any of the schools in the affected list are"
"\A"
"interested in preventing the release of their"
"\A"
"data, please consult with a cyber advisory firm"
"\A"
"and contact us privately at TOX to negotiate a"
"\A"
"settlement. You have till the end of the day by"
"\A"
"12 May 2026 before everything is leaked."
"\A\A"
"Instructure still has until EOD 12 May 2026"
"\A"
"to contact us."
"\A\A"
" \25BC DOWNLOAD AFFECTED_SCHOOLS.TXT \25BC"
"\A"
"91.215.85.103/pay_or_leak/"
"\A"
"instructure_affected_schools_list.txt"
"\A\A"
"visit us: shnyhntww34phqoa6dcgnvps2yu7dlwzmy5"
"\A"
"lkvejwjdo6z7bmgshzayd.onion" !important;
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
https://www.reddit.com/r/Austin/s/UWQvww1dRm
Incidentally I've always hated Canvas and probably every other LMS provider, but what is particularly amusing about this current outage is that it is occurring at exactly the time when universities are demanding that all professors put all of their materials on Canvas, without exception, due to ADA compliance regulations. It is explicitly forbidden for professors to, e.g., refer to pdfs posted on a personal website.
Other commentators here seem not to understand that many faculty also do not enjoy being forced to use Canvas.
And of course the other serious concern I have with Canvas is that they are likely using all the materials faculty upload to train their AI replacements. Many of my colleagues engage in dark humor about this but I haven't noticed much action.
This would be like TurboTax "scheduling maintenance" on April 14th in the US.
There is a lot of people who likely are unaware the latest outage is because they were compromised again.
Them marking the incident as 'Under Maintenance' means the status page isn't reporting this as an outage and adding to downtime%.
No this will not stop this and companies need to be held accountable for their lack of security investment. Every attack should be investigate if the company met an agreed industry standards best practices and staffing, etc. The penalties for not meeting the requirements should be punitive.
2. The payout to the hackers should form part, but not all of the penalties. Pay those guys for their great service to humanity they earned it.
It should be illegal to host insecure services, especially when you're dealing with PII. Breaches keep happening and nobody gives a fuck, because the worst that'll happen is you might lose a handful of customers and buy some "credit monitoring".
Incidents like this should be followed by an audit and charges being laid. Send corp officers to jail for negligent security failures. If you can go to jail for accounting fraud, you should be able to go to jail for cybersecurity-promises-fraud.
They claim to be compliant with a number of security standards [1]. I would love to see a postmortem audit of how much of this they actually implemented.
[1] https://www.instructure.com/en-au/trust-center/compliance
Instead, there should be standard civil penalties for leaking various degrees of PII paid as restitution to the affected individual. Importantly, this must be applied REGARDLESS of "certification" or whether any security practices were "incorrect" or "insufficient". Even if there's a zero-day exploit and you did everything right, you pay. That's the cost of storing people's secrets.
This would make operating services whose whole "thing" is storing a bunch of information about individuals (like Canvas) much more expensive. Good! It's far to cheap to stockpile a ticking time bomb of private info and then walk away paying no damages just because you complied with some out-of-date list of rules or got the stamp of approval from a certification org that's incentivized to give out stamps of approval.
I do agree with the audit and punishments for clear failure to adhere to established standards.
Even if you leave your door unlocked, if someone walks in and steals your stuff, it's a crime. The state has an interest in prosecuting crimes even if the victim didn't do everything they could to prevent it.
That standard is likely to lock people into buying some pretty bad software, but it does little to ensure that they're running reasonably secure systems.
I'm not sure that's a fair analogy.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
Kids from the local uni having a lark, stalkers, vindictive ex employees, local gangs, criminals who understand their victims because they hail from the same community. These are your local hackers. Sift them from the nation states and international crime groups, then deal with the International as a matter of diplomacy. Because we do this so poorly locally, we have little ammunition to when it comes to diplomacy. "reduce attacks by your crime groups and we buy your natural gas, seel you wheat etc"
Want more motivation?- 75% of the local attacks by volume send funds back to terrorist or separatist organizations.
It is not an in-soluble problem. Sentences are a fraction of the answer, effective and receptive reporting processes are more important, then government backing for investigation and enforcement, then policy around home-team activities (ie don't do the bad things yourselves Mr Gov). Deterrence comes after all that.
There are already significant penalties for doing anything like this. The guy involved is in prison for a very long time. I don’t recall the exact number of years but I do remember it was so long that he wasn’t going to see his kids grow up.
I don’t think anyone who puts a little thought into a crime like this doesn’t understand that the penalties are already very huge. You don’t get a slap on the wrist for extorting a company (or person, for that matter)
Really, though, if you want someone to blame, Instructure is not a particularly compelling target. Let's review:
1. Iran is intentionally targeting infrastructure due to a war started by the current administration.
2. China is actively seeking corporate secrets to steal and commercialize for themselves, spurred by extreme protectionism and retaliatory tariffs.
3. North Korea is doing anything they can -- including just taking a remote job by proxy -- in order to extract any money.
4. And Russia is working with and aiding all of them, after everything else going on has forced the embargo to break.
5. All of this while completely alienating every single one of the United States' allies.
6. Meanwhile, the American DHS is currently shut down.
7. And this is after Trump cut funding and personnel for CISA severely enough they've had to end the contract with MS-ISAC, meaning all state and local entities can only remain in the organization if they foot the bill for it directly and CISA and other agencies responsible for cybersecurity are more thinly staffed than they have been in decades.
In short, the current administration systematically disassembled all the protections we have built over the last 100 years, and then placed infrastructure -- schools, in this case, but also power companies, water treatment facilities, communications companies, local governments, hospitals, food producers -- directly on the front lines of the modern geopolitical conflict.
That vast ocean that has kept us safe historically is a poor moat in the modern era.
It's very easy to play with lives that aren't yours.
The best response to a cyberattack on critical systems is to take security seriously. Document the offense, avoid the same mistakes and invest in penetration testing. Of course, nobody is incentivized to do that until they're attacked, so the cycle perpetuates itself.
I think in principle, its sound. Im also just baffled hearing anecdotes from friends that are in big corp world and hearing the type of incidents they have, and how they respond to it.. It makes me wonder if there is enough capable talent to go around for the "boring corp" crowd.
Hint: I don't think there is nearly enough talent to go round, but for these companies, its either that they think they have solid experts (and didn't), OR its not a real priority until you get hit.
An investigative body, the same kind that determines the who, the why, and the how when an airliner crashes or a bridge collapses. Obviously a lot of work needs to be done to get from point A to point B, and it won't happen overnight, but software development is currently a deeply unserious profession and at some point a genuine software engineering practice needs to be developed.
I am, perhaps naively, slightly hopeful that the LLM bullshit plaguing our industry will be the gust of wind needed for the house of cards to collapse and governments to realise that allowing the entire world to be vibe coded is not sustainable.
Aviation’s safety record is not coincidental.
But it turns out that MIT used to have their own homegrown system, and recently switched to Canvas. Bet they're regretting that now.
The build vs. buy decision seems to have swung very hard toward buy in the last decade, and I think that's a shame. Yes, orgs need to focus on their core competency, and sometimes that means outsourcing things that aren't core competencies to third parties. But there are always downsides.
IT staff who are ambitious and talented don’t last long in education. The pay is very low compared to industry. Where I worked, you could retire with a comfortable pension after a number of service years, so the IT staff outsourced as much as possible so they needed to take zero risks to their nest egg. Blame all the problems on the consultants and do as little as possible.
It’s literally where dreams go to die.
MIT is known for the brilliant professors and students but at the end of the day, running a university is pretty standard stuff. They don’t need a genius rockstar to admin the courseware servers.
I'm actually much more interested if there is any financial liability for Instructure here? It's interesting that it's the universities being ransomed, while the technical failure was Instructure's. We're used to uptime SLA's -- what about security breach SLA's?
My guess would be they get likelihood of getting paid when blackmailing 9,000 schools (at least a few would pay up) than blackmailing Canvas/Instructure.
I don't think any SLA/terms would change who gets to feel the pain.
Don't ransom all your eggs in one basket
Also yeah there is value in being able to blame another party, and also being down when everyone else is down.
Edit: https://status.parchment.com/ says "While Canvas, Canvas Beta and Canvas test are currently unavailable, we are simultaneously monitoring all of our other product environments, including Parchment. We continue to see no reason to believe any Parchment resources have been impacted."
do you mean equivalent ?.
I also ran the entire DOMPurify sample XSS and managed to find one way to download custom content onto someone's computer.
1: https://ibb.co/r29RjdnH
We received communication that Canvas is down for "Under Maintenance" although it seems ShineyHunters have compromised Canvas again with that message you posted.
We do not see that message anymore, although all instrucuture.com URLs are down. The list of schools in the ShinyHunters publication can be found here: https://web.archive.org/web/20260507042014/http://91.215.85....
doesn't seem that scheduled to me
That's just the quickest page/status update to throw up; it was a one-liner to push it live back when I was on the deploy rotation.
I'd hazard a guess they have more important things to worry about right now than exact status page messaging ;)
https://news.ycombinator.com/item?id=48025001
I totally understand why a university wouldn’t want to bake their own learning portals but just feels like such a single point of risk to use third party solutions for something like this.
Back in my day… all we had was a school email via on-premise services. I guess we registered for classes in a web portal but that’s about it. The idea of online class was entirely foreign at the time. Ain’t nobody hacking a blue book.
There is a saying in the software security industry that (I'm paraphrasing from rusty memories) a system is secure if the cost of hacking it is higher than the value it protects.
Each system being completely distinct from another means that the cost of hacking the average student goes up by 9000 (from the article, Canvas is used by 9000 schools).
Still not saying that rolling out your own is the preferred solution, but the idea is not as ludicrous as it would seem, and should definitely be entertained and discussed, at least.
As a faculty member at a large university…I have a deep respect for the impossible job of university IT departments.
We originally rolled our on LMS decades ago. When we switched to canvas we kept the home brew running for five years past its expiration date because faculty refused to remove their files. Finally each one was manually moved by IT for the recalcitrant old faculty.
They are large databases yes but they do a lot of small and large things that that analogy glosses over
Well not with that attitude
https://moodle.org/
They used to, in the pre-cloud/SaaS era; and they were much simpler and better UX than the slop that they're renting today, because the actual users were not far from the developers.
The amount of corner cases and performance requirements during rush times (semester start) made it really infeasible for a university to roll their own.
* German universities have this funny system where 51% of such boards are controlled by the professors and the rest is made up of other employees/staff and students. They call it academic participation.
edit: here's the list of impacted universities (unsure if they all have their canvas instances offline, but i'd be surprised if not): http://91.215.85.103/pay_or_leak/instructure_affected_school...
Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.
Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.
It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)
---
Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...
---
(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
dig canvas.ucdavis.edu
dig canvas.duke.eduSomeone dumped the content into a google doc on reddit[1] if anyone's interested.
[1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
Consider surgery instead of software development. There are general best practices, but the difference between a good surgeon and a poor one is a small number of deaths. Malpractice insurance is high. Litigation is constant. And patients still die on the operating table. It's unclear what all the malpractice tort law actually gets you in the end.
In most of these cases, the companies involved did NOT follow standard security practices.
I am pretty sure that is what people mean when they say "held responsible", they mean "held responsible for failing to follow standard security practices", not for the actual act of getting hacked.
Is that really the analogy you want to use the bolster your argument? Licensing was forced on the medical profession because of rampant quackery causing a large number of deaths. Some of the horrors that went on before enforced medical licensing are well-nigh unbelievable, e.g. https://en.wikipedia.org/wiki/John_R._Brinkley
But even if they do everything right, is it really fair to let the companies just shrug their shoulders and say "it happens"? While their users are the ones who really get hurt.
I like this analogy, but deaths shouldn't be the leading indicator just an indicator. Family member had a surgery with well known procedures, say removing a gall bladder. Unfortunately, this surgeon skipped a step in lieu of setting a record for fastest procedure. Because steps were skipped, the gall bladder was not scooped into a net to avoid spilled gall stones which resulted stones spilling into the abdominal cavity requiring numerous follow up surgeries to remove the spilled stones as they made themselves known. So clearly not following accepted procedures should be a clear win in a malpractice case, yeah? Wrong. No doctor would testify against the surgeon and the case was dismissed. I feel like this is exactly how it would work in software security incidents as well.
That was the foundational premise of Dr. Atul Gawande's book The Checklist Manifesto, an expansion of his article The Checklist in The New Yorker [0]
[0] https://www.newyorker.com/magazine/2007/12/10/the-checklist
If we increase the penalties for a company being hacked, you create even MORE incentive for hackers to try to break in, because if they succeed, they have a pretty big stick to threaten companies with when demanding a random payment - not only will the company have the negative effect of the data being leaked and the PR that accompanies it, they now know that if they don't pay and the attack becomes public knowledge, they face a big fine or other punishment.
A company is much more likely to pay a big ransom if they know they are just going to end up paying that much or more in fines if they refuse the ransom and report the hack instead.
If you take this route, and increase punishment for being hacked, you are making a pretty big bet that the main reason companies are hacked is because of poor security practices. I am not sure if that is true or not.
[1] https://digital-strategy.ec.europa.eu/en/policies/cyber-resi... [2] https://ec.europa.eu/commission/presscorner/detail/en/ip_22_...
I'm friends with a professor who complained to me a couple times about how sometimes he will need to scroll through pages and pages of courses he taught in the past. He also mentioned that profs aren't able to delete their own course shells either.
We already bond over how awful the Canvas UX is (and she has a bunch of Chrome extensions to improve it.) Now we’ve got something else to gripe over together.
I'm under the impression files are getting released 12th May. I don't see any reporting on 800GB?
Our whole testing center is down. This is inconvenient, but mainly it's amusing. I swear strangers are talking to each other more. I'm noticing people just sitting in the sun and relaxing. Nature is healing.
(Of course, plenty of people have also just finished their exams, so it's hard to know the cause.)
Any idea what data Instructure-and-also-now-ShinyHunters even purport to have beyond names, profile photos, pronouns, homework assignments, school communications, phone numbers, and email addresses?
i.e. What makes this threat so different from what any old data brokers have already scraped?
What leverage besides aura farming do the ShinyHunters really have?
All I can think of that's really valuable is passwords. And private communications in Canvas DMs. But if you're being at all intimate over your school email, that's kinda on you.
Anyway surely Instructure only stores user public keys or something?
Alternate history question: If they just sold the data, never revealed the hack, and didn't make a scene, from a customer perspective, how different would this be from business as usual?
What we don't have access to includes:
* Already graded work
* Ungraded work
* overall adn assignment grades
* lists of students and student emails from the course
* messages from students that are often sent through gradescope
Just...complete implosion.
I'm honestly surprised more people aren't talking about this.
Shame on your existence basically.
Everything from middle school up to grad school.
It's a particularly interesting time to have this happen too -- many finals going on now.
Of course if you can't complete your exams because of this, that's more of an issue!
They moved away from Teams because it objectively sucked, but I haven't heard of widespread compromises like this in Microsoft's systems so...
And GitHub doesn't provide a way to record grades that remain private per student last I checked, much less sync them to the university, or 99% of other things Canvas does.
I don't love Canvas, but it's far, far preferable to a world without it.
last I checked it appears grades remain private per planet or so ...
Or is it an entirely different class of beast?
Canvas generally is the 'easiest' to use, and the 'cleanest' looking one although D2L Brightspace is pretty good too. Moodle out of the box is pretty confusing and ugly, but I've seen some heavily customized instances that look a lot better. Blackboard is the worst of the bunch IMO.
It's always been as stupid as requiring that your chalkboard, chalk, chairs, bluebooks, pens, paper, gradebook etc etc all come from the same company.
I, for one, am very much looking forward to my IT Gov council meeting tomorrow.
Looking into the payload they sent me this is how they hijacked the screen. Everything in the payload is unchanged except for one line of code:
<link rel="stylesheet" href="https://instructure-uploads.s3.amazonaws.com/account_9363000..." media="all"/>
This links to the following styling sheet:
@import url('https://fonts.googleapis.com/css2?family=Orbitron:wght@500;7...');
html, body { height: 100% !important; overflow: hidden !important; margin: 0 !important; padding: 0 !important; }
body > * { display: none !important; }
body { display: flex !important; align-items: center !important; justify-content: center !important; background: #07080c !important; }
body::before { content: "" !important; position: fixed !important; inset: 0 !important; z-index: 999998 !important; background: radial-gradient(ellipse at 50% 20%, rgba(255,59,59,.06), transparent 55%), radial-gradient(ellipse at 50% 85%, rgba(125,70,152,.04), transparent 45%), repeating-linear-gradient(0deg, rgba(255,255,255,.035), rgba(255,255,255,.035) 1px, transparent 1px, transparent 3px), #07080c !important; pointer-events: none !important; }
body::after { content: "\A\A" "S H I N Y H U N T E R S" "\A" "rooting your systems since '19 ;)" "\A\A\A" "ShinyHunters has breached Instructure (again)." "\A" "Instead of contacting us to resolve it they" "\A" "ignored us and did some \201Csecurity patches\201D." "\A\A" "\26A0 W A R N I N G" "\A\A" "If any of the schools in the affected list are" "\A" "interested in preventing the release of their" "\A" "data, please consult with a cyber advisory firm" "\A" "and contact us privately at TOX to negotiate a" "\A" "settlement. You have till the end of the day by" "\A" "12 May 2026 before everything is leaked." "\A\A" "Instructure still has until EOD 12 May 2026" "\A" "to contact us." "\A\A" " \25BC DOWNLOAD AFFECTED_SCHOOLS.TXT \25BC" "\A" "91.215.85.103/pay_or_leak/" "\A" "instructure_affected_schools_list.txt" "\A\A" "visit us: shnyhntww34phqoa6dcgnvps2yu7dlwzmy5" "\A" "lkvejwjdo6z7bmgshzayd.onion" !important;
}@keyframes pulseWarn { 0% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 50% { box-shadow: 0 0 55px rgba(255,59,59,.4), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } 100% { box-shadow: 0 0 20px rgba(255,59,59,.15), 0 40px 90px rgba(0,0,0,.65), inset 0 0 0 1px rgba(255,255,255,.06); } }
The hack is crude, and it seems unlikely that they have any access to Instructure's developer tools.