Observation of Topological Edge States at the Step Edges on the

Jul 31, 2019 - ... localized around the step edge (see Figure 4 and discussion below), similar to the previous reports on Td-WTe2(20−22) and ZrTe5.(...
0 downloads 0 Views 857KB Size
Subscriber access provided by GUILFORD COLLEGE

Article

Observation of Topological Edge States at the Step Edges on the Surface of Type-II Weyl Semimetal TaIrTe

4

Xu Dong, Maoyuan Wang, Dayu Yan, Xianglin Peng, Ji Li, Wende Xiao, Qinsheng Wang, Junfeng Han, Jie Ma, Youguo Shi, and Yugui Yao ACS Nano, Just Accepted Manuscript • DOI: 10.1021/acsnano.9b04573 • Publication Date (Web): 31 Jul 2019 Downloaded from pubs.acs.org on July 31, 2019

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

Observation of Topological Edge States at the Step Edges on the Surface of Type-II Weyl Semimetal TaIrTe4 Xu Dong†,§, Maoyuan Wang†,§, Dayu Yan‡,§, Xianglin Peng†,§, Ji Li†, Wende Xiao*,†, Qinsheng Wang†, Junfeng Han†, Jie Ma†, Youguo Shi‡, Yugui Yao*,† †Beijing

Key Laboratory of Nanophotonics and Ultrafine Optoelectronic

Systems and Micro-nano Centre, School of Physics, Beijing Institute of Technology, Beijing 100081, China ‡Institute

of Physics, Chinese Academy of Sciences, Beijing 100190, China

ABSTRACT: Topological materials harbor topologically protected boundary states. Recently, TaIrTe4, a ternary transition metal dichalcogenide, was identified as a type-II Weyl semimetal with the minimal nonzero number of Weyl points allowed for a time-reversal invariant Weyl semimetal. Monolayer TaIrTe4 was proposed to host topological edge states, which, however, lacks of experimental evidence. Here, we report on the topological edge states localized at the monolayer step edges of the type-II Weyl semimetal TaIrTe4 using scanning tunneling microscopy. One-dimensional electronic states that show 1 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 2 of 25

substantial robustness against the edge irregularity are observed at the step edges. Theoretical calculations substantiate the topologically non-trivial nature of the edge states and their robustness against the edge termination and layer stacking. The observation of topological edge states at the step edges of TaIrTe4 surfaces suggests that monolayer TaIrTe4 is a two-dimensional topological insulator, providing TaIrTe4 as a promising material for topological physics and devices.

KEYWORDS:

Weyl

semimetal,

two-dimensional

topological

insulator,

topological edge states, scanning tunneling microscopy, scanning tunneling spectroscopy

Topological materials are featured by exotic boundary states.1,2 In a threedimensional (3D) topological insulator (TI), gapless surface states emerge at the material surfaces, while a Weyl semimetal harbors Fermi arc surface states. In a two-dimensional (2D) TI, also known as quantum spin Hall insulator,3,4 onedimensional (1D) conducting edge states propagating along the perimeter of the material can serve as dissipationless spin current channels, which are highly

desirable

for

future

low-power

and

low-dissipation

electronic

applications.5-9

2 ACS Paragon Plus Environment

Page 3 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

In recent years, topological phases in transition metal dichalcogenides (TMDCs) have attracted increasing interests.10-13 One of the most fantastic properties of the layered TMDCs is that their electronic structures can be tuned by the thickness.14-18 For example, bulk Td-WTe2 is a type-II Weyl semimetal,19 while monolayer 1T-WTe2 is a 2D TI with a band gap of ~ 100 meV20-23 and can host the quantum spin Hall effect (QSHE) up to 100 K.24 Notably, as the interlayer van der Waals (vdW) coupling in bulk Td-WTe2 is rather weak, the topological edge states can be preserved at the monolayer step edges of the

Td-WTe2 crystal surfaces.21 Very recently, TaIrTe4, a ternary TMDC with a similar noncentrosymmetric orthorhombic structure to Td-WTe2, was predicted and shortly confirmed by angle-resolved photoemission spectroscopy (ARPES) and transport measurements to be a type-II Weyl semimetal with intriguing topological properties: TaIrTe4 has merely two pairs of Weyl points, the minimal nonzero number of Weyl points allowed for a time-reversal invariant Weyl semimetal, and has the longest Fermi arcs among all the known Weyl semimetals.25-27 The vdW-stacked nature of TaIrTe4 facilitates the preparation of 2D TaIrTe4 with multilayer or even bilayer thickness via mechanical exfoliation.28,29 The 2D TaIrTe4 flakes have good environmental stability28,29 and show the strongest in-plane electrical anisotropy among all the known electrically anisotropic materials with carrier density comparable to good metals as copper and silver.28 Owing to the gapless semimetallic band structure and high responsivity as a result of linear dispersion and suppressed back 3 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 4 of 25

scattering, Lai and coworkers successfully realized broadband, anisotropic and self-powered photodetectors based on 2D TaIrTe4 flakes.29 Monolayer TaIrTe4 was proposed to be a 2D TI,30 which, however, lacks of experimental evidence. It still remains an open question whether topological 1D edge states exist at the step edges of TaIrTe4 surfaces to date. In the present work, we report the direct observation of topological edge states localized at the step edges of the type-II Weyl semimetal TaIrTe4 using low-temperature scanning tunneling microscopy and spectroscopy (LTSTM/STS). By collecting differential conductance (dI/dV) spectra and maps around the atomic step edges on the TaIrTe4 surface, we observe a significant increase of the local density of states (LDOS) at the step edges with respect to the terraces, due to the emergence of 1D topological edge states localized at the step edges. Such topological edge states exhibit substantial robustness against the irregularity of the step edges. Theoretical calculations verify the topologically non-trivial nature of the edge states and their robustness against the edge termination and layer stacking. The observation of topological edge states at the step edges of TaIrTe4 surfaces indicates that monolayer TaIrTe4 is a 2D TI, providing TaIrTe4 as a promising material for exploring topological physics and developing topological devices. RESULTS AND DISCUSSION

4 ACS Paragon Plus Environment

Page 5 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

The TaIrTe4 crystal is comprised of 2D sheets of TaIrTe4 in the a-b plane that are stacked along the c axis via vdW bonding between the layers, as shown in Figure 1a. Each 2D sheet is similar to monolayer 1T' TMDCs, that is, exhibiting 1D zigzag chains along the a axis due to Peierls distortion induced dimerization along the b axis.25 Theoretical calculations reveal lattice constants of a = 3.77 Å, b = 12.42 Å and c = 13.18 Å,25 respectively. It is noteworthy that the ternary TaIrTe4 crystal shares the same space group of Pmn21 with the binary tellurides of WTe2 and MoTe2, but its lattice constant along the b axis is doubled as it has two metallic elements.

Figure 1. Topography of TaIrTe4. (a) Crystal structure of TaIrTe4. (b) STM image of the TaIrTe4 (001) surface showing a step running along the a axis with a monolayer height. Scanning parameters: sample bias (Vs) = -0.5 V, tunneling 5 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 6 of 25

current (It) = 40 pA, measure temperature (T) = 77 K. The lattice axes are indicated by the black arrows. (c) Atomic-resolution STM image (Vs = -0.8 V, It = 130 pA, T = 77 K) of the TaIrTe4 (001) surface. The unit cell is indicated by the white rectangle. (d) Atomic-resolution STM image (Vs = -0.80 V, It = 75 pA,

T = 77 K) showing a Te vacancy (marked by the white circle) at the surface. (e) Simulated STM image (left) and structural model of the TaIrTe4 (001) surface (right). TaIrTe4 single crystals were grown by a solid-state reaction (see the Methods). After cleaving in ultrahigh vacuum, clean TaIrTe4 (001) surfaces can be easily obtained. Figure 1b displays an STM topography of such a surface. Steps with various heights can be observed on the as-prepared TaIrTe4 (001) surface (see Figure S1 in the Supporting Information). Line profile analysis (Figure S1) reveals a minimum step height of 6.6 ± 0.1 Å, consistent with the spacing between the adjacent TaIrTe4 layers. The atomically ordered terraces exhibit bright stripes parallel to each other with a separation of 12.4 ± 0.1 Å, in good agreement with the lattice constant along the b axis. High-resolution STM images (Figure 1c,d) clearly resolve the top-layer Te atoms of the TaIrTe4 (001) surface. A lattice length of 3.8 ± 0.1 Å is measured along the 1D chains, in line with theoretical calculations. The surface Te atoms exhibit four different apparent heights, as they are bonded with different metal elements and/or with different bond lengths (Figure S2). The simulated STM image (Figure 1e) based

6 ACS Paragon Plus Environment

Page 7 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

on density functional theory (DFT) calculations reproduces the experiments very well and reveals that the surface Te atoms, each bonded with two Ta atoms and an Ir atom, appear as the brightest protrusion in STM images. Individual depressions can also be observed on the surface (Figure 1d and Figure S1). Each of these depressions is assigned to a surface Te vacancy.

Figure 2. Edge states of TaIrTe4 around straight step edges. (a) STM image (Vs = -0.4 V, It = 60 pA, T = 5 K) of a straight step edge running along the a direction with a monolayer height. (b) dI/dV spectra measured at the positions (marked by blue, red and green dots) along the straight line perpendicular to the step edge in (a). (c) dI/dV spectra measured at the terrace center (blue curve) and at the step edge (red curve). (d) Calculated LDOS of the terrace (blue curve) and the step edge (red curve). (e) STM Topography of another straight

7 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 8 of 25

monolayer step edge (Vs = -0.1 V, It = 100 pA, T = 5 K). (f-h) dI/dV maps of the same area of (e) taken at various sample bias.

To ascertain the electronic structures at the step edges, we have acquired a series of dI/dV spectra across a monolayer straight step edge running along the

a direction, as shown in Figure 2a. Each individual spectrum is vertically offset with equal intervals of 2 pS for a direct comparison of the conductance. Evidently, all spectra have a finite conductance for all energies (Figure 2b), in accord with the semimetallic feature of TaIrTe4. The spectra collected on the bright strips are slightly different than that at the grooves between the strips (Figure 2b), as the surface Te atoms are bonded to different metal elements. The spectra nearly stay the same as the ones measured on the wide terrace when approaching the step edge from the lower terrace (Figure 2b, green curves). However, when approaching the step edge from the upper terrace, the tunneling conductance is profoundly enhanced near the step edge in the region of 30 ~ 100 mV and show a broad peak centered at ~ 60 mV (Figure 2b, red curves). These behaviors suggest the existence of prominent edge states. For a direct comparison, the spectra taken at the wide terrace (blue curve) and the step edge (red curve) are plotted in Figure 2c, where the peak at ~ 60 mV is conspicuous in the spectrum acquired at the step edge. Figure 2d shows the calculated DOS of the bulk (blue curve) and monolayer-height step edge (red 8 ACS Paragon Plus Environment

Page 9 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

curve). An additional peak at ~ 50 meV is obvious in the DOS of the step edge, in good agreement with the experiments. Careful analysis reveals that these electronic states are topologically protected 1D edge states localized around the step edge (see Figure 4 and discussion below), similar to the previous reports on Td-WTe220-22 and ZrTe5.31,32 To straightforwardly illustrate the spatial distribution of the edge states, we have measured dI/dV maps around another monolayer straight step edge running along the a direction (Figure 2e). The maps at 100 mV and 50 mV (Figure 2f,g) exhibit similar conductance enhancement at the step edge, but it is rather homogenous at 0 mV (Figure 2h), as the energy is out of the window of the edge states. These behaviors confirm the localization of the edge states along the step edge and unveil their physical origin and 1D character. We note that the step edges are formed by breaking the atomic bonds in the TaIrTe4 layers during sample cleaving. The resulted step edge can exhibit different atomic terminations (Figure S2), local relaxations, reconstructions and adsorbates, etc. The detailed spectroscopic shape of the topological edge states may vary at different locations of the step edge, resulting in the conductance fluctuations of the edge states intensity and thus the inhomogenous appearance in the dI/dV map shown in Figure 2f,g. Similar observations were reported in the topological edge states of many other topological materials.21,31,32 It is also noteworthy that individual protrusions on

9 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 10 of 25

the upper terrace can be observed in the dI/dV map taken at 50 mV (Figure 2g), but the same positions show depressions in the dI/dV maps taken at 100 mV (Figure 2f) and 0 mV (Figure 2h). These features can be assigned to the defects of the sample.

Figure 3. Edge states of TaIrTe4 around irregular step edges. (a) STM image (Vs = -0.4 V, It = 60 pA, T = 5 K) of an irregular shaped step edge with a monolayer height. (b) dI/dV spectra collected at the locations (marked by blue, red and green dots) with different distance away from the step edge in (a). (c) dI/dV spectra acquired at the locations (indicated by magenta dots) along the step edge in (a). (d) STM Topography of another irregular step edge (Vs = -0.1 V, It = 100 pA, T = 5 K). (e-g) dI/dV maps of the same area of (d) taken at various sample bias. 10 ACS Paragon Plus Environment

Page 11 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

As the edge states of TaIrTe4 are topologically protected, they must be insensitive to the detailed geometry or the atomic structures of the step edges. To check the robustness of the edge states, we have also acquired a series of dI/dV spectra along and across an irregular step edge with a monolayer height, as shown in Figure 3a. Each spectrum is vertically offset with equal intervals of 2 pS for a direct comparison of the conductance, as plotted in Figure 3b,c. It is clearly seen that only the conductance near the step edge exhibits significant enhancement between ~ 30 mV and ~ 100 mV when approaching the step edge from the upper terrace (Figure 3b, red curves), while the spectra nearly stay the same upon approaching the step edge from the lower terrace (Figure 3b, green curves), again indicating the existence of edge states. Notably, the spectra measured along this irregular step edge exhibit distinct line shapes (Figure 3c, magenta curves). The conductance fluctuation suggests that the cleavage induced step edge has diverse local geometry and/or chemistry at different positions. Nevertheless, all spectra collected at different positions of the step edge exhibit significant conductance enhancement between ~ 30 mV and ~ 100 mV (Figure 3c, magenta curves). The dI/dV maps (Figure 3e,f) of a monolayerheight irregular step edge (Figure 3d) exhibit identical energy dependence as the straight step edge in Figure 2e. These behaviors indicate that the edge states are very robust against the variation of local geometry and/or chemical environment.7, 31-33 The robustness of the edge states confirms their topological

11 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 12 of 25

nature and excludes the possibility of trivial origin from the termination of atomic lattices.

Figure 4. Calculated edge states of TaIrTe4. (a) Band structure of continuous monolayer TaIrTe4. (b) Band structure of bilayer TaIrTe4. (c) Band structure of semi-infinite monolayer TaIrTe4. (d) Band structure of TaIrTe4 with a step constructed from a semi-infinite monolayer TaIrTe4 supported by a continuous monolayer TaIrTe4. (e,f) Side and top views of charge density distribution of the topological edge states, respectively. To further illuminate the topological nature of the observed edge states, we have performed theoretical calculations. The calculated band structure (Figure 4a) clearly show that freestanding monolayer TaIrTe4 is a 2D TI with a gap of 28 meV, in good agreement with previous report.30 For TaIrTe4 with increasing thickness ≥ 2 layers, the band gap is closed (Figure 4b) and it becomes a 12 ACS Paragon Plus Environment

Page 13 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

semimetal due to the interlayer vdW coupling.30 Figure 4c shows the calculated band structure of a semi-infinite monolayer TaIrTe4 with an edge. The emergence of Dirac-like topological edge states around the Gamma (G) point demonstrates that monolayer TaIrTe4 is a 2D TI, in agreement with previous report.30 To compare with the experiments, we mimic the step edges at the surfaces of TaIrTe4 crystal using the same semi-infinite monolayer TaIrTe4 but supported by a continuous monolayer TaIrTe4. As plotted in Figure 4d, the Dirac-like topological step edge states around the G point are essentially the same as those from the freestanding semi-infinite monolayer TaIrTe4. The weak interlayer vdW coupling leads to the expected evolution of TaIrTe4 from 2D TI to semimetal.30 Nevertheless, the topological edge states are still preserved. Moreover, our calculations show that the topologically protected edge states are insensitive to the local structures of the step edges (Figure S3 and S4). The calculated charge density of the edge states (Figure 4e and Figure S5) is essentially localized along the step edge, in good agreement with the experiments. Therefore, the calculations strongly support the topological origin of the edge states observed in our STM/STS experiments. CONCLUSIONS In summary, we have studied the electronic structures of the step edges of the TaIrTe4 (001) surface by LT-STM/STS combined with theoretical calculations. We demonstrate the existence of 1D topological edge states residing at the 13 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 14 of 25

step edges, evidencing that monolayer TaIrTe4 is a 2D TI. The topologically protected edge states are robust against the irregularity of the step edges and the layer stacking. As the vdW-stacked nature of TaIrTe4 facilitates the preparation of 2D TaIrTe4 via mechanical exfoliation, it is very promising to obtain monolayer TaIrTe4 and fabricate monolayer TaIrTe4-based devices for QSHE and optoelectronics. METHODS Growth of TaIrTe4 Crystals: Single crystals of TaIrTe4 were grown by a solidstate reaction with the help of Te flux. High-purity (≥ 99.99%) Ta and Ir powders and Te pieces were mixed and loaded in an alumina crucible in a molar ratio of 1:1:10. The crucible was sealed in a vacuum quartz tube. The quartz tube was slowly heated up to 1373 K, maintained for 10 hours and slowly cooled down to 873 K at a rate of 2 K/h. The shiny, needle-shaped single crystals were separated from the Te flux by centrifuging. STM/STS Measurements: The experiment was carried out in an ultrahigh vacuum (base pressure of 1 × 10-10 mbar) LT-STM system, equipped with standard surface processing facilities. TaIrTe4 single crystals were cleaved in

situ under ultrahigh vacuum condition at room temperature. The freshly cleaved surface was transferred into the STM chamber immediately and slowly cooled down to 77 or 5 K. The STM measurements were performed with electrochemically etched tungsten tips, which were calibrated with respect to 14 ACS Paragon Plus Environment

Page 15 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

the Au(111) surface state prior to spectroscopic measurements. The images were taken in constant-current mode at 77 K or 5 K, and all given voltages are referred to the sample. Spectroscopic measurements were performed at 5 K by using a lock-in technique with a 5 mVrms sinusoidal modulation at a frequency of 677 Hz. First-Principles Calculations: The theoretical simulations are based on DFT calculation, which were performed using the projector augmented wave34 method implemented in VASP35 with Perdew-Burke-Ernzerhof (PBE) parameterization of GGA36 functional using 300 eV energy cutoff of the plane wave basis and a 14 × 4 × 1 k-points mesh. The structure was optimized with the vdW correction (SCAN+rVV1037-39) until the force on each atom was less than 0.01 eV/ Å. For band gap calculation, we used hybrid functional HSE06.40,41 The parameters to calculate surface states of the materials were obtained from Wannier90 code42 with VASP2WANNIER9043 interface. To calculate the band structures, we used surface Green function method44 to calculate the spectral function A(ε,k) = −1/ Im[TrGr(ε,k)] (See Supporting Informatino for details).

ASSOCIATED CONTENT Supporting Information. 15 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 16 of 25

The Supporting Information is available free of charge on the ACS Publications website. Figures of large-scale topography for TaIrTe4; figure of detailed structural model for monolayer TaIrTe4; figures of band structures for monolayer TaIrTe4 with different edge termination; figures of band structures for bilayer TaIrTe4 with different edge termination; figure of charge density distribution of the topological edge states of a semi-infinite monolayer TaIrTe4; details about the method for band structure (spectral function) calculations.

AUTHOR INFORMATION Corresponding Author * E-mail: [email protected]. * E-mail: [email protected]. Author Contributions §X.D.,

M.Y.W., D.Y.Y. and X.L.P. contributed equally to this work.

ACKNOWLEDGMENT This work was financially supported by the National Science Foundation of China (Grants No. 51661135026, 21773008, 11734003 and 11774399) and the 16 ACS Paragon Plus Environment

Page 17 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

National Key Research and Development Program of China (Grant No. 2016YFA0300904 and 2016YFA0300600).

REFERENCES 1. Kane, C. L.; Mele, E. J., Quantum Spin Hall Effect in Graphene. Phys. Rev.

Lett. 2005, 95, 226801. 2. Bernevig, B. A.; Hughes, T. L.; Zhang, S. C., Quantum Spin Hall Effect and Topological Phase Transition in HgTe Quantum Wells. Science 2006, 314, 1757-1761. 3. Hasan, M. Z.; Kane, C. L., Colloquium: Topological Insulators. Rev. Mod.

Phys. 2010, 82, 3045-3067. 4. Qi, X.-L.; Zhang, S.-C., Topological Insulators and Superconductors. Rev.

Mod. Phys. 2011, 83, 1057-1110. 5. Liu, C. C.; Feng, W.; Yao, Y., Quantum Spin Hall Effect in Silicene and TwoDimensional Germanium. Phys. Rev. Lett. 2011, 107, 076802. 6. Xu, Y.; Yan, B.; Zhang, H. J.; Wang, J.; Xu, G.; Tang, P.; Duan, W.; Zhang, S. C., Large-Gap Quantum Spin Hall Insulators in Tin Films. Phys. Rev. Lett. 2013, 111, 136804.

17 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 18 of 25

7. Yang, F.; Miao, L.; Wang, Z. F.; Yao, M. Y.; Zhu, F.; Song, Y. R.; Wang, M. X.; Xu, J. P.; Fedorov, A. V.; Sun, Z.; Zhang, G. B.; Liu, C.; Liu, F.; Qian, D.; Gao, C. L.; Jia, J. F., Spatial and Energy Distribution of Topological Edge States in Single Bi(111) Bilayer. Phys. Rev. Lett. 2012, 109, 016801. 8. Roth, A.; Brune, C.; Buhmann, H.; Molenkamp, L. W.; Maciejko, J.; Qi, X. L.; Zhang, S. C., Nonlocal Transport in the Quantum Spin Hall State. Science 2009, 325, 294-297. 9. Roy, R., Topological Phases and the Quantum Spin Hall Effect in Three Dimensions. Phys. Rev. B: Condens. Matter Mater. Phys. 2009, 79, 195322. 10.

Qian, X.; Liu, J.; Fu, L.; Li, J., Quantum Spin Hall Effect in Two-

Dimensional Transition Metal Dichalcogenides. Science 2014, 346, 1344-1347. 11.

Cazalilla, M. A.; Ochoa, H.; Guinea, F., Quantum Spin Hall Effect in

Two-Dimensional Crystals of Transition-Metal Dichalcogenides. Phys. Rev.

Lett. 2014, 113, 077201. 12.

Deng, K.; Wan, G.; Deng, P.; Zhang, K.; Ding, S.; Wang, E.; Yan, M.;

Huang, H.; Zhang, H.; Xu, Z.; Denlinger, J.; Fedorov, A.; Yang, H.; Duan, W.; Yao, H.; Wu, Y.; Fan, S.; Zhang, H.; Chen, X.; Zhou, S., Experimental Observation of Topological Fermi Arcs in Type-II Weyl Semimetal MoTe2. Nat.

Phys. 2016, 12, 1105-1110.

18 ACS Paragon Plus Environment

Page 19 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

13.

Nie, S. M.; Song, Z.; Weng, H.; Fang, Z., Quantum Spin Hall Effect in

Two-Dimensional Transition-Metal Dichalcogenide Haeckelites. Phys. Rev. B:

Condens. Matter Mater. Phys. 2015, 91, 235434. 14.

Zheng, F.; Cai, C.; Ge, S.; Zhang, X.; Liu, X.; Lu, H.; Zhang, Y.; Qiu, J.;

Taniguchi, T.; Watanabe, K.; Jia, S.; Qi, J.; Chen, J. H.; Sun, D.; Feng, J., On the Quantum Spin Hall Gap of Monolayer 1T'-WTe2. Adv. Mater. 2016, 28, 4845-4851. 15.

Naylor, C. H.; Parkin, W. M.; Ping, J.; Gao, Z.; Zhou, Y. R.; Kim, Y.;

Streller, F.; Carpick, R. W.; Rappe, A. M.; Drndic, M.; Kikkawa, J. M.; Johnson, A. T., Monolayer Single-Crystal 1T'-MoTe2 Grown by Chemical Vapor Deposition Exhibits Weak Antilocalization Effect. Nano Lett. 2016, 16, 42974304. 16.

Lin, Y.-C.; Dumcenco, D. O.; Huang, Y.-S.; Suenaga, K., Atomic

Mechanism of the Semiconducting-to-Metallic Phase Transition in SingleLayered MoS2. Nat. Nanotechnol. 2014, 9, 391-396. 17.

Zhang, C.; Kc, S.; Nie, Y.; Liang, C.; Vandenberghe, W. G.; Longo, R.

C.; Zheng, Y.; Kong, F.; Hong, S.; Wallace, R. M.; Cho, K., Charge Mediated Reversible Metal-Insulator Transition in Monolayer MoTe2 and WxMo1-xTe2 Alloy. ACS Nano 2016, 10, 7370-7375.

19 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

18.

Page 20 of 25

Keum, D. H.; Cho, S.; Kim, J. H.; Choe, D.-H.; Sung, H.-J.; Kan, M.;

Kang, H.; Hwang, J.-Y.; Kim, S. W.; Yang, H.; Chang, K. J.; Lee, Y. H., Bandgap Opening in Few-Layered Monoclinic MoTe2. Nat. Phys. 2015, 11, 482-486. 19.

Soluyanov, A. A.; Gresch, D.; Wang, Z.; Wu, Q.; Troyer, M.; Dai, X.;

Bernevig, B. A., Type-II Weyl Semimetals. Nature 2015, 527, 495-498. 20.

Tang, S.; Zhang, C.; Wong, D.; Pedramrazi, Z.; Tsai, H.-Z.; Jia, C.;

Moritz, B.; Claassen, M.; Ryu, H.; Kahn, S.; Jiang, J.; Yan, H.; Hashimoto, M.; Lu, D.; Moore, R. G.; Hwang, C.-C.; Hwang, C.; Hussain, Z.; Chen, Y.; Ugeda, M. M.; et al., Quantum Spin Hall State in Monolayer 1T'-WTe2. Nat. Phys. 2017,

13, 683-687. 21.

Peng, L.; Yuan, Y.; Li, G.; Yang, X.; Xian, J. J.; Yi, C. J.; Shi, Y. G.; Fu,

Y. S., Observation of Topological States Residing at Step Edges of WTe2. Nat.

Commun. 2017, 8, 659. 22.

Jia, Z.-Y.; Song, Y.-H.; Li, X.-B.; Ran, K.; Lu, P.; Zheng, H.-J.; Zhu, X.-

Y.; Shi, Z.-Q.; Sun, J.; Wen, J.; Xing, D.; Li, S.-C., Direct Visualization of a TwoDimensional Topological Insulator in the Single-Layer 1T'-WTe2. Phys. Rev. B:

Condens. Matter Mater. Phys. 2017, 96, 041108. 23.

Fei, Z.; Palomaki, T.; Wu, S.; Zhao, W.; Cai, X.; Sun, B.; Nguyen, P.;

Finney, J.; Xu, X.; Cobden, D. H., Edge Conduction in Monolayer WTe2. Nat.

Phys. 2017, 13, 677-682. 20 ACS Paragon Plus Environment

Page 21 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

24.

Wu, S.; Fatemi, V.; Gibson, Q. D.; Watanabe, K.; Taniguchi, T.; Cava,

R. J.; Jarillo-Herrero, P., Observation of the Quantum Spin Hall Effect Up to 100 Kelvin in a Monolayer Crystal. Science 2018, 359, 76-79. 25.

Koepernik, K.; Kasinathan, D.; Efremov, D. V.; Khim, S.; Borisenko, S.;

Buchner, B.; van den Brink, J., TaIrTe4: A Ternary Type-II Weyl Semimetal.

Phys. Rev. B: Condens. Matter Mater. Phys. 2016, 93, 201101. 26.

Haubold, E.; Koepernik, K.; Efremov, D.; Khim, S.; Fedorov, A.;

Kushnirenko, Y.; van den Brink, J.; Wurmehl, S.; Buchner, B.; Kim, T. K.; Hoesch, M.; Sumida, K.; Taguchi, K.; Yoshikawa, T.; Kimura, A.; Okuda, T.; Borisenko, S. V., Experimental Realization of Type-II Weyl State in Noncentrosymmetric TaIrTe4. Phys. Rev. B: Condens. Matter Mater. Phys. 2017, 95, 241108. 27.

Belopolski, I.; Yu, P.; Sanchez, D. S.; Ishida, Y.; Chang, T. R.; Zhang,

S. S.; Xu, S. Y.; Zheng, H.; Chang, G.; Bian, G.; Jeng, H. T.; Kondo, T.; Lin, H.; Liu, Z.; Shin, S.; Hasan, M. Z., Signatures of a Time-Reversal Symmetric Weyl Semimetal with Only Four Weyl Points. Nat. Commun. 2017, 8, 942. 28.

Liu, Y.; Gu, Q.; Peng, Y.; Qi, S.; Zhang, N.; Zhang, Y.; Ma, X.; Zhu, R.;

Tong, L.; Feng, J.; Liu, Z.; Chen, J. H., Raman Signatures of Broken Inversion Symmetry and In-Plane Anisotropy in Type-II Weyl Semimetal Candidate TaIrTe4. Adv. Mater. 2018, 30, e1706402.

21 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

29.

Page 22 of 25

Lai, J.; Liu, Y.; Ma, J.; Zhuo, X.; Peng, Y.; Lu, W.; Liu, Z.; Chen, J.; Sun,

D., Broadband Anisotropic Photoresponse of the "Hydrogen Atom" Version Type-II Weyl Semimetal Candidate TaIrTe4. ACS Nano 2018, 12, 4055-4061. 30.

Liu, J.; Wang, H.; Fang, C.; Fu, L.; Qian, X., van der Waals Stacking-

Induced Topological Phase Transition in Layered Ternary Transition Metal Chalcogenides. Nano Lett. 2017, 17, 467-475. 31.

Li, X. B.; Huang, W. K.; Lv, Y. Y.; Zhang, K. W.; Yang, C. L.; Zhang, B.

B.; Chen, Y. B.; Yao, S. H.; Zhou, J.; Lu, M. H.; Sheng, L.; Li, S. C.; Jia, J. F.; Xue, Q. K.; Chen, Y. F.; Xing, D. Y., Experimental Observation of Topological Edge States at the Surface Step Edge of the Topological Insulator ZrTe5. Phys.

Rev. Lett. 2016, 116, 176803. 32.

Wu, R.; Ma, J. Z.; Nie, S. M.; Zhao, L. X.; Huang, X.; Yin, J. X.; Fu, B.

B.; Richard, P.; Chen, G. F.; Fang, Z.; Dai, X.; Weng, H. M.; Qian, T.; Ding, H.; Pan, S. H., Evidence for Topological Edge States in a Large Energy Gap near the Step Edges on the Surface of ZrTe5. Phys. Rev. X 2016, 6, 021017. 33.

Reis, F.; Li, G.; Dudy, L.; Bauernfeind, M.; Glass, S.; Hanke, W.;

Thomale, R.; Schafer, J.; Claessen, R., Bismuthene on a SiC Substrate: A Candidate for a High-Temperature Quantum Spin Hall Material. Science 2017,

357, 287-290.

22 ACS Paragon Plus Environment

Page 23 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

34.

Kresse, G.; Joubert, D., From Ultrasoft Pseudopotentials to the

Projector Augmented-Wave Method. Phys. Rev. B: Condens. Matter Mater.

Phys. 1999, 59, 1758-1775. 35.

Kresse, G.; Furthmüller, J., Efficiency of Ab-initio Total Energy

Calculations for Metals and Semiconductors Using a Plane-Wave Basis Set.

Comput. Mater. Sci. 1996, 6, 15-50. 36.

Perdew, J. P.; Burke, K.; Ernzerhof, M., Generalized Gradient

Approximation Made Simple. Phys. Rev. Lett. 1996, 77, 3865-3868. 37.

Sabatini, R.; Gorni, T.; de Gironcoli, S., Nonlocal van der Waals Density

Functional Made Simple and Efficient. Phys. Rev. B: Condens. Matter Mater.

Phys. 2013, 87, 041108. 38.

Sun, J.; Ruzsinszky, A.; Perdew, J. P., Strongly Constrained and

Appropriately Normed Semilocal Density Functional. Phys. Rev. Lett. 2015,

115, 036402. 39.

Sun, J.; Remsing, R. C.; Zhang, Y.; Sun, Z.; Ruzsinszky, A.; Peng, H.;

Yang, Z.; Paul, A.; Waghmare, U.; Wu, X.; Klein, M. L.; Perdew, J. P., Accurate First-Principles Structures and Energies of Diversely Bonded Systems From an Efficient Density Functional. Nat. Chem. 2016, 8, 831-836. 40.

Heyd, J.; Scuseria, G. E.; Ernzerhof, M., Hybrid Functionals Based on

a Screened Coulomb Potential. J. Chem. Phys. 2003, 118, 8207-8215. 23 ACS Paragon Plus Environment

ACS Nano 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

41.

Page 24 of 25

Krukau, A. V.; Vydrov, O. A.; Izmaylov, A. F.; Scuseria, G. E., Influence

of the Exchange Screening Parameter on the Performance of Screened Hybrid Functionals. J. Chem. Phys. 2006, 125, 224106. 42.

Mostofi, A. A.; Yates, J. R.; Lee, Y.-S.; Souza, I.; Vanderbilt, D.;

Marzari, N., Wannier90: A Tool for Obtaining Maximally-Localised Wannier Functions. Comput. Phys. Commun. 2008, 178, 685-699. 43.

Franchini, C.; Kováčik, R.; Marsman, M.; Murthy, S. S.; He, J.; Ederer,

C.; Kresse, G., Maximally Localized Wannier Functions in LaMnO3 within PBE + U, Hybrid Functionals and Partially Self-Consistent GW: An Efficient Route to Construct Ab initio Tight-Binding Parameters for eg Perovskites. J.

Phys.: Condens. Matter 2012, 24, 235602. 44.

Sancho, M. P. L.; Sancho, J. M. L.; Sancho, J. M. L.; Rubio, J., Highly

Convergent Schemes for the Calculation of Bulk and Surface Green Functions.

J. Phys. F: Met. Phys. 1985, 15, 851-858.

24 ACS Paragon Plus Environment

Page 25 of 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

TOC graphic:

25 ACS Paragon Plus Environment