\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}



\usepackage{float}








\begin{document}

\title{The European Factor: From ARM to~Atos}



\author[1]{Anne C. Elster}%
\affil[1]{Affiliation not available}%


\vspace{-1em}



  
  \date{June 15, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup








\sloppy


It has been both a strange and unsettling~ year with a lot of
uncertainties related to COVID-19. Both the~ premier supercomputing
conferences ISC'20 and SC20 went virtual,~ preventing users from
exploring the show floors with their examples of novel architectures and
seeing the demos of their related tools. Companies have also slowed
their procurement of high-end systems during 2020, as was discussed
during the presentation of top500.org statistics at the Top-500
Birds-of-a-Feather session at SC20. The Top-500 list is a measurement of
the world's largest computers using the High-Performance Linpack
benchmark (see~\href{http://www.top500.org}{www.top500.org}). Systems
must solve large dense systems of linear equations without using the
operation-saving Strassen algorithm or a mix of lower precision, but
performing the calculations conforming to LU factorization with partial
pivoting using~~\( 2/3 n^3+O(n^2)\) double-precision (64-bit)~ floating
point numbers.

Despite less turnover of the world's largest computer systems,
technologies continue to advance, and the trends are more interesting
and diverse than ever.

The US dominated the~ computing industry---including the supercomputer
industry---for years, until Japan gave the US a jolt with the
introduction of the Earth Simulator back in 2002.~ This put
supercomputing back in the spotlight, especially in the US, but also in
Europe. For the past decade, the US, Japan and China have~ dominated at
the top of the Top-500 list.~ As can be seen from the following table,
these systems were built primarily on US-designed chips from IBM and
Intel, coupled with accelerators from NVIDIA.~ The AMD-based
\emph{Titan} Cray/HPE system and the \emph{Sunway} system in China, as
well as the new ARM-based Fujitsu \emph{Fugaku} supercomputer, are
notable exceptions. In this article, we will discuss the history of ARM
and its current impact on HPC as well as some other European trends.

\par\null\selectlanguage{english}
\begin{table}[H]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CCCCC}
Name & Dates no. 1 & Where & Architecture & Cores \\
Tianhe-1A & 2010-11-01 & China & NUDT TH MPP,\selectlanguage{ngerman}  Intel Xeon X5670 @ 2.93 GHz, Infiniband FDR FT-1000, Nvidia 2050 GPUs & 186 368 \\
K-Computer & June 2011 - Nov 2011 & Japan & Fujitsu SPARC64 VIIIfx @ 2.0GHz with\selectlanguage{ngerman}  Tofu interconn. & 705 024 \\
Sequoia & 2012-06-01 & US & IBM BlueGene/Q, IBM Power BQC\selectlanguage{ngerman}  16C @ 1.6GHz, Custom interconnect & 1 572 864 \\
Titan & 2012-11-01 & US & Cray/HPE Cray XK7, \textbf{AMD Opteron} 6274 16C @ 2.2GHz w/~ Cray
Gemini interconn., Nvidia K20x GPUs
 & 560 640 \\
Tianhe-2A & June 2013- Nov. 2015 & China & Intel Xeon, TH Express-2, Matrix 2000 & 4 981 760 \\
Sunway TaihuLight & Jun 2016 - Nov 2017 & China & Sunway MPP, Sunway SW26010 260C, NRCPC & 10 649 600 \\
Summit & June 2018- Nov 2019 & US & IBM AC922 Power9 + Nvidia GV100 GPUs, Mellanox & 2 424 492 \\
Fugaku SC & June 2020- Nov. 2020 & Japan & Fujitsu/RIKEN \textbf{ARM-based A64FX 48C} @ 2.2 GHz w/ Tofu
interconnects
 & 7 299 072 (Jun) \\
 &  &  &  &  \\
 &  &  &  & 7 630 848 (Nov) \\
\end{tabulary}
\caption{{Top500 No. 1 systems 2010-20
{\label{809478}}%
}}
\end{table}\par\null

\section*{ARM ---from RISC and Embedded systems to
Supercomputers}

{\label{259215}}

ARM is~a 30-year old UK-based semiconductor IP (Intellectual Property)
company that started as Advanced RISC Machines Ltd., a joint venture
that included former UK company Acorn Computers, Apple Computers (now
Apple Inc.) ~\hyperref[csl:1]{(\textit{{Apple to Join Acorn, VLSI in Chip-Making Venture}}, n.d.)}, and VLSI Technology (a US company later
bought by Phillips for USD 1 billion and now part of the Philips
spin-off NXP Semiconductors). ARM was bought in 2016 by the Japanese
conglomerate and investing company Softbank Group, which also has a main
stake in US Sprint and also has significant T-Mobile stock via the
merger of Sprint and T-Mobile that was completed in April 2020.~ In
September~2020, NVIDIA announced that it is buying
ARM~\hyperref[csl:2]{(\textit{{NVIDIA to Acquire Arm for USD 40 Billion}}, n.d.)}, but it may take time to complete since it is
running into some licensing issues regarding its subsidiary ARM - China.

Unlike Intel, AMD, and Freescale, ARM does not produce any computer
chips, but licenses its architecture designs and product ecosystem to
other vendors.~It has particularly focused on low-power CPUs and GPUs
for mobile and embedded devices, and has over the last several years
acquired dozens of smaller companies, such as Falanx, a Norwegian
startup that is now known as ARM Norway and is behind their Mali GPU
processor.~

ARM now offers a wide range of IP design products, including CPUs, GPUs
and microcontrollers. ARM CPUs and NPUs (Neural Processing Units)
include Cortex-A, Cortex-M, Cortex-R, Neoverse, Ethos, and SecureCore.
Mobile companies such as Apple and Samsung~often license the designs
from ARM to manufacture and integrate them into their own System-on-chip
(SoC) designs with other components such as GPUs (sometimes ARM's Mali)
or modem/radio baseband components (for mobile phones). In fact, due to
the proliferation of mobile and handheld devices such as tablets, ARM
was claimed to have 75\% of the world's CPU market in 2005. As the
demand for processing power of these devices has increased so has the
demand for their CPUs; since 2018, ARM has also offered server-class CPU
designs. If NVIDIA's purchase of ARM goes through, it will be very
interesting to see how it impacts future ARM designs that are already
influenced by AI workloads.

\par\null

\subsection*{ARM-based Astra}

{\label{636057}}

The first ARM-based supercomputer to appear on the the Top-500 list was
\emph{Astra} at Sandia National Labs,~announced at SC18. The HPE
(Hewlett Packard Enterprise)-built system ranked at number 204 with
1.529 petaFLOPS, and made number 36 on the HP Conjugate Gradient
benchmark, a benchmark more targeted to its use. \emph{Astra} was the
first system which that was part of the US DOE NNSA (National Nuclear
Security Administration) Vanguard program that looks at prototype
systems for advanced architectures.~

One of the \emph{Astra} nodes is about 100 times faster than the
ARM-based CPU chips found in cell phones. The \emph{Astra} system has
5,184 ARM-based Cavium ThunderX2~ 28-core processors. Cavium Inc. grew
from designing networking processors to itself being bought in 2018 by
Marvell Technology group, a Bermuda-registered conglomerate.

Michael Aguilar, the~main systems administrator of \emph{Astra}, gave a
nice introduction to the system in 2019 at Stanford~\hyperref[csl:3]{(\textit{{ASTRA: A Large Scale ARM64 HPC Deployment - insideHPC}}, n.d.)}.~
He describes how they have ported many of the familiar packages ranging
from~ OpenMPI to UCX, where one of the interesting challenges they faced
was that, unlike Intel cores, the ARM cores did not appear next to each
other, so they behaved in a very non-uniform fashion. Socket-direct, a
new feature on the architecture, seems to alleviate many performance
issues associated with core locality, but requires a new awareness at
this level. They have had good results with the system so far, achieving
from 1.6X speedup on Monte Carlo to 1.87X on linear solver versus the
Intel Haswell-based \emph{Trinity} ASC platform they used as a baseline.

However, as far as power consumption, they did not see it particularly
more efficient than the Intel ones, but have not been focusing on this
either.

\par\null

\section*{Fugaku and ARM}

{\label{121320}}

In Summer 2020, the new Japanese~\emph{Fugaku} Supercomputer at RIKEN
gained the top spot on the Top-500 list. Named ``High Peak''~ after Mt
Fuji,~ its ARM-based A64FX 48C chips are jointly developed by~ RIKEN and
Fujitsu and uses Fujitsu's Tofu (torus fusion) interconnects.~ The A64
chip is a TSMC 7nm FinFET with 8.8 billion transistors and 594 signal
pins. It also has SVE 512 x 2 vector extensions~\hyperref[csl:4]{(\textit{{Supercomputer Fugaku : Fujitsu Global}}, n.d.)}.~

Note that one of the biggest differences between x86 chips from Intel
and AMD, from a vector operation point of view, is that modern Intel
chips have one 512-bit AVX2 vector unit, whereas AMD chips have 2-3
128-bit units.

\emph{Fugaku} maintained its number-1 spot in November
2020~\hyperref[csl:5]{(\textit{{Japan's Fugaku keeps position as fastest supercomputer}}, n.d.)}, but had, unlike the 3 next systems
(\emph{Summit, Sierra,} and~\emph{Sunway}), added over 300,000~ cores
(from 7,299,072 cores to 7,630,848 cores). Its A64 chip has in addition
to the 48-core compute unit, 2 or 4 assist units that run the OS
(operating system).

The~\emph{Fugaku} system is also very energy efficient, ranking number
10 on the Green500 list, unlike~\emph{Summit, Sierra,} and
Sunway.~~\emph{Selene}, the Nvidia DGX A100 system at NVIDIA with
555,520 cores, ranked in Nov. 2020 at number 5 both on the Top-500 and
Green500 lists.

The Fujitsu A64fx is also featured in Stony Brook's HPE Apollo 80/ Cray
computer~\emph{Ookami} (``wolf'' in Japanese), announced in Nov. 2020 .
(Cray was formally acquired by HPE in Sept. 2019.) The \emph{Ookami}
system is the first system announced outside Japan that will feature the
A64fx.

\par\null

\section*{Atos and AMD EPYC}

{\label{240112}}

Another interesting entry on the Nov. 2020 Top-500 list is the number-7
\emph{JUWELS Booster Module}, an Atos Bull Sequena XH200 based on the
AMD EPYC chips installed at Forschungzentrum J\selectlanguage{ngerman}ülich (FZJ) in Germany.~

Atos is a French company that emerged from the merger of two French
companies and a Dutch IT company. It has since grown with the
merger/acquisition of several companies, including Siemens IT (2011),
Bull, and Xerox ITO (2014), the latter a well-known HPC vendor in
Europe. It has also strengthened its US presence through the acquisition
of Xerox ITO (2014) and Syntel (2018).~

As a historical note, Heide~\hyperref[csl:6]{(Heide, 1991)}~\hyperref[csl:7]{(Heide, 2008)}
describes how both Bull and IBM have their roots dating back to
punch-card machines. Bull is in fact named after the Norwegian pioneer,
Fredrik Rosing Bull who in 1918 started to build punched-card machines.
Bull died in 1924, but his work was continued by others in Norway, and
later (1931) it was relocated to Paris in France where it, after several
reorganizations, emerged as Bull, a subsidiary of Atos Technologies.

In May 2020 Atos announced that it was adding the new NVIDIA A100 GPUs
to the Jülich system~\hyperref[csl:8]{(\textit{{Atos Launches First Supercomputer Equipped with NVIDIA A100 GPU - Atos}}, n.d.)}, which clearly contributed to
its number 7 Nov. 2020 ranking on the Top-500.~

\par\null

\section*{The Nanometer Race ---Intel versus
TSMC}

{\label{269210}}

Currently, just three chip manufacturers in the world are using less
than 14nm node technologies: Intel, TSMC~ (Taiwan Semiconductor
Manufacturing Company), and Samsung. China's SMIC has yet to do smaller
than 14nm processes. The nm (nanometer)~denomination previoulsy referred
to gate length, but has in the last several years been tied to
``technology node'' size and become more vendor-dependent.~ For
instance, Intel's 10nm nodes are assumed to be compatible with
foundries' 7nm nodes.

\emph{So how is this affecting Supercomputing?}

\emph{Fugaku}~ and AMD's recent EPYC 7nm chip are both manufactured at
TSMC, which also produces the new 5nm M1 chip for Apple. However, TSMC
reports it is struggling to meet the demand for the Apple chips, so
Apple may have to again also use Samsung.

Intel announced through its earnings statement in July 2020 that it is
having issues with its 7nm manufacturing, thus causing an expected
1-year or more delay of its GPU Ponte Vecchio chips. This may again
delay the~\emph{Aurora} system planned at Argonne National
Labs~\hyperref[csl:9]{(\textit{{Intel delays 7nm, considers using rival foundries}}, n.d.)}~\hyperref[csl:10]{(\textit{{AMD-Powered Frontier May Become First US Exascale Supercomputer Due to Intel's 7nm Process Issues}}, n.d.)}, one of the two new exascale
systems planned in the US in 2021. The AMD-powered~\emph{Frontier}
system at Oak Ridge National Laboratory is supposedly still on schedule,
and will likely become the first exascale system in the US.

Of course, if we allow for low- or mixed-precision and look at the
HPL-AI benchmarks---differing from the Linpack benchmarks used for the
Top-500 rankings that only allow for full 64-bit precisions---we are
already in the exascale era.~\emph{Fugaku} also ranked number 1 on this
benchmark in Nov. 2020 peaking at 2.0 exaFLOPS~\hyperref[csl:11]{(\textit{{John Boyd, IEEE Spectrum Tech Talk, June 2020:Full Page Reload}}, n.d.)}.

\par\null

\section*{EuroHPC JU and the next wave of European
supercomputers}

{\label{711068}}

Europe have had some systems in the top ten of the Top-500 list,
including the current Atos systems, but is now pushing hard for their
new~ EuroHPC JU (European HPC Joint Undertaking) with a roadmap that
includes the following systems:

\textbf{Pre-Exascale class:}

\begin{itemize}
\tightlist
\item
  \emph{LUMI} (Large Unified Modern Infrastructure) at CSC's center in
  Kajaani, Finland~\hyperref[csl:12]{(\textit{{LUMI}}, n.d.)}~\hyperref[csl:13]{(\textit{{EuroHPC Exec. Dir. Talks Procurement, EPI, and Europe’s Efforts to Control its HPC Destiny}}, n.d.)}
\item
  \emph{Marenostrum 5} at Barcelona Supercomputing Centre, Spain
\item
  \emph{Leonardo} at CINECA, Italy~\hyperref[csl:14]{(\textit{{CINECA Partners with Atos for 240PFLOPS ‘Leonardo’ Supercomputer - insideHPC}}, n.d.)}
\end{itemize}

\par\null

\textbf{Petascale supercomputers}:

\begin{itemize}
\tightlist
\item
  LuxProvide, Luxembourg~
\item
  IZUM, Slovenia
\item
  IT4Innovations National Supercomputing Centre, Czech Republic
\item
  Sofiatech, Bulgaria
\item
  Minho Advanced Computing Centre (MACC), Portugal,
\end{itemize}

\emph{LUMI} will be an HPE Cray EX supercomputer with AMD 64-core EPYC
CPUs and feature the new AMD Instinct GPUs, and is projected to be
faster than the current number 1~\emph{Fugaku}~supercomputer.
And~\emph{Leonardo} is now announced to be a 240-petaFLOPS Atos'
BullSequana XH2000 with ParTec's ParaStation Modulo
software.~\emph{Marenostrum 5}'s design has not been announced as of
this article's writing.

\textbf{FPGA ---finally maturing for HPC?}

Field-Programmable Gate Arrays (FPGAs) that let you encode algorithms
and functions in hardware offer great potential speeds for user
application codes, but have been, and still are, fairly hard to program
for scientists. However, two of the word's largest CPU makers have
bought FPGA companies (AMD announced in October 2020 it will be buying
the FPGA company~ Xilinx Inc.). Although Xilinx is~ headquartered in San
Jose, it has a strong presence in Europe and Asia, and has also started
to offer FPGA-based accelerator cards targeting AI and cloud workloads.
The deal is expected to go through by the end of 2021. Xilinx´s new data
center accelerator card, the Alveo FPGA U250, targets AI/ML, graphics
computing and HPC computations~\hyperref[csl:15]{(\textit{{Alveo U250 Data Center Accelerator Card}}, n.d.)}.

FPGAs and other emerging accelerators will be discussed in a future
department feature.

\selectlanguage{english}
\FloatBarrier
\section*{References}\sloppy
\phantomsection
\label{csl:1}\url{https://www.latimes.com/archives/la-xpm-1990-11-28-fi-4993-story.html.} \url{https://www.latimes.com/archives/la-xpm-1990-11-28-fi-4993-story.html}

\phantomsection
\label{csl:2}\url{https://www.globenewswire.com/news-release/2020/09/13/2092654/0/en/NVIDIA-to-Acquire-Arm-for-40-Billion-Creating-World-s-Premier-Computing-Company-for-the-Age-of-AI.html.} \url{http://www.globenewswire.com/news-release/2020/09/13/2092654/0/en/NVIDIA-to-Acquire-Arm-for-40-Billion-Creating-World-s-Premier-Computing-Company-for-the-Age-of-AI.html}

\phantomsection
\label{csl:3}\url{https://insidehpc.com/2019/02/astra-a-large-scale-arm64-hpc-deployment/.} \url{https://insidehpc.com/2019/02/astra-a-large-scale-arm64-hpc-deployment/}

\phantomsection
\label{csl:4}\url{https://www.fujitsu.com/global/about/innovation/fugaku/.} \url{https://www.fujitsu.com/global/about/innovation/fugaku/}

\phantomsection
\label{csl:5}\url{https://www.japantimes.co.jp/news/2020/11/17/national/japan-fugaku-fastest-supercomputer/.} \url{https://www.japantimes.co.jp/news/2020/11/17/national/japan-fugaku-fastest-supercomputer/}

\phantomsection
\label{csl:6}{From Invention to Production: The Development of Punched-card Machines by F. R. Bull and K. A. Knutsen 1918-1930}. (1991). \textit{{IEEE} Annals of the History of Computing}, \textit{13}(3), 261–272. \url{https://doi.org/10.1109/mahc.1991.10024}

\phantomsection
\label{csl:7}{Punched cards for professional European offices: revisiting the dynamics of information technology diffusion from the United States to Europe 1889{\textendash}1918}. (2008). \textit{History and Technology}, \textit{24}(4), 307–320. \url{https://doi.org/10.1080/07341510802044686}

\phantomsection
\label{csl:8}\url{https://atos.net/en/2020/press-release/general-press-releases_2020_05_14/atos-launches-first-supercomputer-equipped-with-nvidia-a100-gpu.} \url{https://atos.net/en/2020/press-release/general-press-releases_2020_05_14/atos-launches-first-supercomputer-equipped-with-nvidia-a100-gpu}

\phantomsection
\label{csl:9}\url{https://www.datacenterdynamics.com/en/news/intel-delays-7nm-considers-using-rival-foundries/.} \url{https://www.datacenterdynamics.com/en/news/intel-delays-7nm-considers-using-rival-foundries/}

\phantomsection
\label{csl:10}\url{https://phonemantra.com/amd-powered-frontier-may-become-first-us-exascale-supercomputer-due-to-intels-7nm-process-issues/.} \url{https://phonemantra.com/amd-powered-frontier-may-become-first-us-exascale-supercomputer-due-to-intels-7nm-process-issues/}

\phantomsection
\label{csl:11}\url{https://spectrum.ieee.org/tech-talk/computing/hardware/japans-fugaku-supercomputer-is-first-in-the-world-to-simultaneously-top-all-high-performance-benchmarks.} \url{https://spectrum.ieee.org/tech-talk/computing/hardware/japans-fugaku-supercomputer-is-first-in-the-world-to-simultaneously-top-all-high-performance-benchmarks}

\phantomsection
\label{csl:12}\url{https://www.lumi-supercomputer.eu/lumi-one-of-the-worlds-mightiest-supercomputers/value} here

\phantomsection
\label{csl:13}\url{https://www.hpcwire.com/2020/11/19/eurohpc-exec-dir-talks-procurement-epi-and-europes-efforts-to-control-its-hpc-destiny/value} here

\phantomsection
\label{csl:14}\url{https://insidehpc.com/2020/11/cineca-partners-with-atos-for-240pflops-leonardo-supercomputer/.} \url{https://insidehpc.com/2020/11/cineca-partners-with-atos-for-240pflops-leonardo-supercomputer/}

\phantomsection
\label{csl:15}\url{https://www.xilinx.com/products/boards-and-kits/alveo/u250.html.} \url{https://www.xilinx.com/products/boards-and-kits/alveo/u250.html}

\end{document}

