I've had the most success with describing concurrency and parallelism as follows:
The term "Parallelism" these days is synonymous with "CPU Parallelism." Parallelism has existed for a long time. For example, network cards have been able to do I/O parallelism for seemingly forever. Could you imagine if a computer could only have a single network request running at a time? Therefore, what parallelism (read: CPU Parallelism) means is CPU execution happening at literally the same time. Emphasis on literally.
Concurrency is how we make sense of and communicate the above mess. In the days of single cores, we mostly cared about concurrency as a way to communicate I/O parallelism in a manageable way. Now we use concurrency to express both forms of parallelism in our programs. Good implementations of concurrency (Erlang) make all of this transparent to the user. Bad implementations of concurrency (Node.js) have the user doing all of the work (callbacks).
Pretty much everything, honestly. Erlang was built from the start to accommodate massive concurrency and robustness requirements, in particular for telecom switches, but largely applicable to anything.
Node's a single-threaded reactor pattern event loop, whereas Erlang/OTP is an SMP-enabled virtual machine with a preemptive scheduler that manages lightweight actors called processes, all isolated from each other with their own stack and heap, that can be cheaply spawned and communicate only via message passing. On top of that, Erlang offers location transparency for processes and easy node distribution out of the box. OTP then provides a framework for fault tolerance and reusable components by supplying generic behaviors (server, FSM, etc.), supervisors for monitoring and restarting processes when they crash, endpoints for hot code reloading, and more.
You should try it one day. There's really nothing else like it, even if its underlying concepts aren't necessarily novel - but they're well combined and largely obscure in mainstream practice. A lot of platforms have tried to implement their own equivalent actor model approaches (Celluloid for Python, Akka for Scala...), but none reach the same level.
> all isolated from each other with their own stack and heap,
That is really a unique property of the BEAM VM. Most languages built on it Erlang, Elixir, LFE, Joxa, etc can take advantage of that.
Often when telling someone technical about how you can have processes that behave like OS processes -- fully isolated heap, and if one crashes it doesn't affect the others, and which are also very lightweight -- only a few KB of memory overhead, they think there is some trick or misunderstanding. Like "sure but they are just green threads" or "they can't be just a few KB of memory, processes have to be bigger than that".
Isn't there a saying something to the effect of "an advanced enough technology looks to many as indistiguishable from magic".
Thanks for the overview! I have seen a lot of hype around Elixir lately, do you happen to know if it contains all of the benefits of Erlang? Coming from a Ruby background, Elixir looks very nice to work with syntax-wise compared to Erlang.
You have all of the benefits of Erlang the language, Erlang the VM and Erlang libraries. Elixir adds hygienic macros, an awesome package manager (hex), task/test runner and build tool (mix). Elixir also has a Ruby-ish syntax, but I leave it up to the reader to make their own opinions of Elixir syntax vs Erlang syntax as that is not a war I'm interested in.
Erlang has schedulers (one per CPU) that schedules when code gets run. If an Erlang process (Erlang's unit of computation/concurrency) is tied up doing IO, the scheduler simply jumps to another Erlang process. In Node.js you write out callbacks/promises/soup du jour that get run after some specified thing completes. In Erlang your code looks completely sequential.
And on Node.js, whatever thing you run probably gets put into a bog-standard thread pool. So all that's really happening is that your IO requests are getting sent to a limited number of long-running threads. Run enough operations at the same time and you can tie up the thread pool, which most people don't even know exists. In some ways, it's the worst of both worlds: it has the limitations of a thread pool with the API complexity of a callback system.
Couldn't that be said for nearly every programming language that doesn't share an underlying VM? I'm not sure what you're getting at. Do you mind expanding on that?
The best description of concurrency vs parallelism I've heard is from "Clojure for the Brave and True" where it's described via a Lady Gaga song, pure genius: http://www.braveclojure.com/concurrency/
When I actually looked up these terms I found no strong, exact agreement on meanings.
However, of the definitions I did find, the ones referenced in the Clojure documentation seemed most reasonable - from memory, something like the difference between coordination of simultaneous tasks, and their execution.
This definition allows for the existence of concurrent systems without formal concurrency primitives - every action game deals with race conditions even though the gameplay loop is typically single threaded, because it has to code apparently independent agents against a global shared-state environment, and the order in which these agents process data matters. In most instances there is no crisis for the game if the situation is left poorly specified - it may slightly bias some elements or defer events by an additional frame - but for certain things it can blow up into a showstopper bug with physics or cutscene triggers, or harm responsiveness.
This correlates with a common trope from students who encounter the agents problem for the first time - they try to run each AI on a thread, further pushing away their ability to direct concurrency, while "gaining" parallel performance(not really, of course).
Concurrency is a possible feature of an abstract machine, exposed as a programming-language-design paradigm. A language can target a platform that "has concurrency", and this heavily shapes the language. You can end up "coding Go/Erlang in C"—writing concurrent code in your head for a concurrent abstract machine, then translating it to your target language.
Parallelism, meanwhile, is a VM/runtime/platform implementation detail.
You can have concurrency with one CPU but you can't have parallelism with one CPU.
Concurrency is the ordering and dependency (or lack of) between two (or more) paths of executions. Parallelism is the doing of work in parallel. They describe two different things. A program can be either concurrent only, parallel only, or both, and there's where the confusion come in.
You're technically right (multi-core CPUs can do true parallelism), but you were downvoted probably because your point was obvious (to the people who downvoted you).
A multi-core CPU is basically multiple CPUs. The details (multicores share the same die, use the same caches, etc.) are mostly irrelevant here.
Multithreading on a single CPU doesn't have parallelism. You are just switching between tasks very fast serially.
A multi-core CPU for all intents and purposes has multiple CPU's. For the discussion on concurrency and parallelism, the distinction is on one execution unit vs multiple execution units.
I saw this image [1] a while ago. At first I thought it was clever, implying that multithreaded programming often seems like chaos in practice. But actually, that chaos is a result of the nondeterministic nature of concurrent processing. Each thread is doing it's own thing as quickly as possible. Specifically, it is important to note that concurrent threads cannot be expected to perform at equal speed.
In this particular analogy, if the goal is to consume the food as quickly as possible, this may be the most efficient solution.
Concurrency: you have at least two threads sharing control of one CPU. Only one thread runs at a time. There is no parallelism. The important thing is that multiple threads are running.
Parallelism: the two threads are running simultaneously on two CPUs. There is concurrency too since there are multiple threads.
Oh, look, another group of people making another arbitrary distinction between these two words, which is shared by no one else.
In grad school, we used concurrency to mean interleaved, single processor threading, while parallelism meant multiprocessor.
I've heard other people use this distinction to mean fork-join style concurrency vs. shared memory parallelism.
Can we just accept that these two words mean roughly the same thing and stop trying to get one community's niggling technical distinction to be accepted as gospel?
I've seen this distinction made in many "communities" including Ruby, Python and Perl.
Perhaps in your domain/language this distinction isn't important but it is for some of us. The patterns you use to work with concurrent vs parallel code differ, which means it is an important distinction to understand.
My claim isn't that this distinction isn't useful, it's that the terminology is a disaster, and we should find better terminology to make this distinction. There's no consistency in using these particular terms to refer to these particular meanings.
Every time this article or a similar subject is brought up, the thread quickly fills up with people all giving different definitions of concurrency and parallelism.
Nobody can agree on them so let's just assume they mean the same thing and move to more interesting discussions, shall we?
> so let's just assume they mean the same thing and move to more interesting discussions, shall we?
I don't think it is that complicated.
Concurrency is a property of the algorithm or problem space. Things can happen independent of each other. Two or more client requests can read a file. Logically they don't have to wait for each other. There is no sequential dependence. These could be represented in code as two threads, two futures, promises, Goroutines, Tasks, callback chains, actors etc <insert favorite concurrency primitive>.
Parallelism is a property of the execution environment. "Can the concurrent parts of the program execute at the same time?" The machine happens to have 2 CPU cores and two requests arriving at the same time can execute at the some time. Or it has only one CPU and even though they can execute at the same time, they don't. One might be blocked waiting for another or they might be preempted by the scheduler every milliseconds, execute a bit, then switch to another.
EDIT: Here is Joe Armstrong's interpretation. Which is similar to this and includes a cute diagram with coffee machines:
> let's just assume they mean the same thing and move to more interesting discussions
They don't mean the same thing, and that is a poisonous attitude to have. Parallelism is about performance. Concurrency is about synchronization. The difference might not matter to you, but for certain things (programming language design, library design, system architecture) it is paramount.
One thing nearly everyone agrees on: a single-core processor cannot do parallelism, but it's perfectly capable of concurrency. How do you reconcile that with your view that they are the same?
The difference matters very much to me, what is silly is that nobody agrees on which one is concurrency and which one is parallelism. In this thread alone, I found two people with a different definition from yours.
Furthermore, for the many conversations I've had on the topic (and I've had hundreds), I don't recall a single instance where someone used the wrong term and it caused the slightest bit of confusion.
While I agree with you, it is better to focus on practical things other than pedantic names, you can really notice the difference more when you look at the foundation of Go's language level concurrency paradigm.
Concurrency in Go is based on Communicating Sequential Processes (CSP) by Hoare; this was originally published in 1978. Reading CSP you see all the ideas are abstractly formalized in math terms: there really is no notion of multiple streams of execution occurring at the same time (i.e., multiple hardware threads). So, a distinction is made between things cooperating (concurrency in their parlance) and things running at the same time (parallelism).
To continue this idea, parallelism, again using their terms, really is an implementation of the more abstract concept of concurrency. That is, given a description of concurrent processes communicating you can run them in parallel (and reason about their correctness).
If you do not make the distinction, regardless of the terms you use, you remove the layers of abstraction necessary to reason about complex systems (separation of high level concepts and their implementation).
> Nobody can agree on them so let's just assume they mean the same thing and move to more interesting discussions, shall we?
Concurrency and parallelism need different abstractions; try using actors for parallelism or MPI for concurrency! If they mean the same thing, how do we distinguish between them as is absolutely necessary?
Erlang is about concurrency, not parallelism; ditto for Go Channels. MPI was designed for parallelism in the HPC sector, not concurrency for distributed servers.
Saying concurrency and parallelism are equivalent basically invalidates all design decisions made by these systems.
Those are not either or. Concurrency sits at a higher level of abstraction -- at the problem domain. "I'll create an actor for this request, another actor for that, maybe another one for monitoring". These are concurrent units.
Parallelism happens if my machine has more than one CPU and these concurrency units will actaully execute at the same time. Or maybe I have a single CPU only. I'll have not parallelism. They'll be sequenced and scheduled in and out of the single CPU in some small or larger time slices. But they'll not execute in parallel.
So maybe I am running my machine in 1989 (Erlang was doing work back then too) with only a single CPU. I still want to create many concurrent units of work. Or maybe I am running in 2015 and I have 64 core machine and I specifically choose Erlang because I expect to get lots of parallelism for cheap as my concurrent units are largely independent of each other.
It really depends on what you are doing. If you are processing requests concurrently, parallelism is a nice benefit, but the requirement is to do things at the same time correctly with resource contentions involved. Parallelism is more about breaking a job into bits so it goes faster...e.g. what is done in the HPC and Big Data worlds with MPI, CUDA, MapReduce, ...
Concurrency is necessary property of the work as defined, parallelism is an opportunistic property. Confusing the two concepts can lead to people trying to do HPC work with Erlang.
If you are working on an HPC workload, you wouldn't choose Erlang, but you might get a happy speed up for a standard distributed application. Parallelism is strictly about performance, concurrency is about correctness. When you go off and do a HPC or big data program, you might play with MPI, CUDA, MapReduce, or maybe just roll something yourself. No one goes...we should do that in Erlang! Likewise, when you go off and do a concurrent distributed system, Actors make a lot of sense given the concurrency requirements involved (who would think of using CUDA for that?).
Perhaps the problem is not about concurrency and parallelism, but different kinds of computations.
It's a semantic debat, but you seem to redefine parallelism as HPC. I respectfully disagree, parallelism is not necessarily about getting faster, it is really about actually executing computations simultaneously.
Erlang uses parallelism for fault-tolerance purposes, because you need several physical machines running in parallel on different sites to achieve outstanding fault-tolerance. MPI/CUDA/MapReduce uses parallelism for performance reasons.
I do not know if it is true as I couldn't find the source, but I remember reading that aircrafts run all computations in parallel on three different architectures (Power and two others), using three different algorithms, to ensure that the results are not compromised either by a faulty algorithm, or a CPU bug. In that case parallelism is used for safety.
> Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel").
...
Parallel computer programs are more difficult to write than sequential ones,[5] because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common.
And the article for concurrency [2]:
> In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other. The computations may be executing on multiple cores in the same chip, preemptively time-shared threads on the same processor, or executed on physically separated processors.
Concurrency is a problem (doing things at the same time leads to resource contention), while parallelism is a performance technique. Fault tolerance is something else altogether, and can apply to concurrency problems as well as parallel speedups. Doing things in parallel as a form of redundancy isn't really parallelism...in general, the word parallelism != "in parallel" while concurrency != "concurrently," though they are definitely related.
> Confusing the two concepts can lead to people trying to do HPC work with Erlang.
You're right that Erlang is not the best choice for HPC, but you're wrong about the motive. There's nothing inherently inefficient about the actor model. Erlang's low performance comes from unrelated technical choices (bytecode running on a VM, shitty string handling, etc.).
The actor model itself is designed for concurrency, but it is close enough to message passing that people often confuse both, and there are lots of points in between that can be considered actorish (just not Hewitt's original model).
Shared memory is also quite nice when you can afford it in non-distributed settings (never use MPI on just one machine!). If your problem can fit in GPU memory, CUDA will do wonders.
Is chapel still around? I thought the Darpa money ran out.
Not sure if troll, but... Just because the majority cannot agree does not mean the terms are not defined. Concurrency and parallelism are quite distinct concepts (though not entirely) [1,2,3].
> Just because the majority cannot agree does not mean the terms are not defined
I never said that, just that a lot of people have different definitions for these words, all contradicting each other. So in effect, these words have become meaningless.
Several cars on a highway moving in one direction.
On a one lane highway several cars can move forward, one after the other concurrently.
On a two lane highway on each lane several cars can move forward one after the other concurrently AND on the other lane other cars can move in parallel to them.
___
One CPU (core) having one program counter pc executes one instruction after an other and keeps track of the position in the instruction current by increasing the pc. With special code it is possible to make the pc jump to an completely unrelated code block and later make it jump back to the first code block. The core sees and executes one current of instructions but the code-blocks, the intentions of the code-blocks can be quite unrelated. There can be several concurrent code-blocks managed (usually by the OS, Kernel) interlaced into the instruction current the CPU core executes.
If there are several cpu execution cores several instruction currents with their own interlaced code-blocks can be processed in parallel.
This insistence on terminology is sadly incorrect, and should be: multitasking isn't parallel processing.
Concurrency and parallelism are very general, vague terms that mean approximately the same thing: processing of some unspecified kind happening simultaneously.
For instance, parallelism can be in a single thread: "instruction-level parallelism" (pipelines in a CPU: fetching some instructions while others are being decoded and others still are executing on multiple functional units).
I don't get the difficulty here. Contextual synonyms are similar. And different. Running concurrently with someone else is not the same as running parallel with them. Though, to go parallel with them, you likely were running concurrently. At least in most contexts.
The term "Parallelism" these days is synonymous with "CPU Parallelism." Parallelism has existed for a long time. For example, network cards have been able to do I/O parallelism for seemingly forever. Could you imagine if a computer could only have a single network request running at a time? Therefore, what parallelism (read: CPU Parallelism) means is CPU execution happening at literally the same time. Emphasis on literally.
Concurrency is how we make sense of and communicate the above mess. In the days of single cores, we mostly cared about concurrency as a way to communicate I/O parallelism in a manageable way. Now we use concurrency to express both forms of parallelism in our programs. Good implementations of concurrency (Erlang) make all of this transparent to the user. Bad implementations of concurrency (Node.js) have the user doing all of the work (callbacks).