diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2016-10-07 20:19:31 -0700 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2016-10-07 20:19:31 -0700 |
| commit | 07021b43597f506cc525d139ed1a94e79cf184f2 (patch) | |
| tree | 888ab33ec69b397ae6f8a2c82d14197047ee827e | |
| parent | d1f5323370fceaed43a7ee38f4c7bfc7e70f28d0 (diff) | |
| parent | b7b7013cac55d794940bd9cb7b7c55c9dececac4 (diff) | |
Merge tag 'powerpc-4.9-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux
Pull powerpc updates from Michael Ellerman:
"Highlights:
- Major rework of Book3S 64-bit exception vectors (Nicholas Piggin)
- Use gas sections for arranging exception vectors et. al.
- Large set of TM cleanups and selftests (Cyril Bur)
- Enable transactional memory (TM) lazily for userspace (Cyril Bur)
- Support for XZ compression in the zImage wrapper (Oliver
O'Halloran)
- Add support for bpf constant blinding (Naveen N. Rao)
- Beginnings of upstream support for PA Semi Nemo motherboards
(Darren Stevens)
Fixes:
- Ensure .mem(init|exit).text are within _stext/_etext (Michael
Ellerman)
- xmon: Don't use ld on 32-bit (Michael Ellerman)
- vdso64: Use double word compare on pointers (Anton Blanchard)
- powerpc/nvram: Fix an incorrect partition merge (Pan Xinhui)
- powerpc: Fix usage of _PAGE_RO in hugepage (Christophe Leroy)
- powerpc/mm: Update FORCE_MAX_ZONEORDER range to allow hugetlb w/4K
(Aneesh Kumar K.V)
- Fix memory leak in queue_hotplug_event() error path (Andrew
Donnellan)
- Replay hypervisor maintenance interrupt first (Nicholas Piggin)
Various performance optimisations (Anton Blanchard):
- Align hot loops of memset() and backwards_memcpy()
- During context switch, check before setting mm_cpumask
- Remove static branch prediction in atomic{, 64}_add_unless
- Only disable HAVE_EFFICIENT_UNALIGNED_ACCESS on POWER7 little
endian
- Set default CPU type to POWER8 for little endian builds
Cleanups & features:
- Sparse fixes/cleanups (Daniel Axtens)
- Preserve CFAR value on SLB miss caused by access to bogus address
(Paul Mackerras)
- Radix MMU fixups for POWER9 (Aneesh Kumar K.V)
- Support for setting used_(vsr|vr|spe) in sigreturn path (for CRIU)
(Simon Guo)
- Optimise syscall entry for virtual, relocatable case (Nicholas
Piggin)
- Optimise MSR handling in exception handling (Nicholas Piggin)
- Support for kexec with Radix MMU (Benjamin Herrenschmidt)
- powernv EEH fixes (Russell Currey)
- Suprise PCI hotplug support for powernv (Gavin Shan)
- Endian/sparse fixes for powernv PCI (Gavin Shan)
- Defconfig updates (Anton Blanchard)
- KVM: PPC: Book3S HV: Migrate pinned pages out of CMA (Balbir Singh)
- cxl: Flush PSL cache before resetting the adapter (Frederic Barrat)
- cxl: replace loop with for_each_child_of_node(), remove unneeded
of_node_put() (Andrew Donnellan)
- Fix HV facility unavailable to use correct handler (Nicholas
Piggin)
- Remove unnecessary syscall trampoline (Nicholas Piggin)
- fadump: Fix build break when CONFIG_PROC_VMCORE=n (Michael
Ellerman)
- Quieten EEH message when no adapters are found (Anton Blanchard)
- powernv: Add PHB register dump debugfs handle (Russell Currey)
- Use kprobe blacklist for exception handlers & asm functions
(Nicholas Piggin)
- Document the syscall ABI (Nicholas Piggin)
- MAINTAINERS: Update cxl maintainers (Michael Neuling)
- powerpc: Remove all usages of NO_IRQ (Michael Ellerman)
Minor cleanups:
- Andrew Donnellan, Christophe Leroy, Colin Ian King, Cyril Bur,
Frederic Barrat, Pan Xinhui, PrasannaKumar Muralidharan, Rui Teng,
Simon Guo"
* tag 'powerpc-4.9-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux: (156 commits)
powerpc/bpf: Add support for bpf constant blinding
powerpc/bpf: Implement support for tail calls
powerpc/bpf: Introduce accessors for using the tmp local stack space
powerpc/fadump: Fix build break when CONFIG_PROC_VMCORE=n
powerpc: tm: Enable transactional memory (TM) lazily for userspace
powerpc/tm: Add TM Unavailable Exception
powerpc: Remove do_load_up_transact_{fpu,altivec}
powerpc: tm: Rename transct_(*) to ck(\1)_state
powerpc: tm: Always use fp_state and vr_state to store live registers
selftests/powerpc: Add checks for transactional VSXs in signal contexts
selftests/powerpc: Add checks for transactional VMXs in signal contexts
selftests/powerpc: Add checks for transactional FPUs in signal contexts
selftests/powerpc: Add checks for transactional GPRs in signal contexts
selftests/powerpc: Check that signals always get delivered
selftests/powerpc: Add TM tcheck helpers in C
selftests/powerpc: Allow tests to extend their kill timeout
selftests/powerpc: Introduce GPR asm helper header file
selftests/powerpc: Move VMX stack frame macros to header file
selftests/powerpc: Rework FPU stack placement macros and move to header file
selftests/powerpc: Check for VSX preservation across userspace preemption
...
227 files changed, 5444 insertions, 2904 deletions
diff --git a/Documentation/powerpc/syscall64-abi.txt b/Documentation/powerpc/syscall64-abi.txt new file mode 100644 index 000000000000..fa716a0d88bd --- /dev/null +++ b/Documentation/powerpc/syscall64-abi.txt @@ -0,0 +1,105 @@ +=============================================== +Power Architecture 64-bit Linux system call ABI +=============================================== + +syscall +======= + +syscall calling sequence[*] matches the Power Architecture 64-bit ELF ABI +specification C function calling sequence, including register preservation +rules, with the following differences. + +[*] Some syscalls (typically low-level management functions) may have + different calling sequences (e.g., rt_sigreturn). + +Parameters and return value +--------------------------- +The system call number is specified in r0. + +There is a maximum of 6 integer parameters to a syscall, passed in r3-r8. + +Both a return value and a return error code are returned. cr0.SO is the return +error code, and r3 is the return value or error code. When cr0.SO is clear, +the syscall succeeded and r3 is the return value. When cr0.SO is set, the +syscall failed and r3 is the error code that generally corresponds to errno. + +Stack +----- +System calls do not modify the caller's stack frame. For example, the caller's +stack frame LR and CR save fields are not used. + +Register preservation rules +--------------------------- +Register preservation rules match the ELF ABI calling sequence with the +following differences: + +r0: Volatile. (System call number.) +r3: Volatile. (Parameter 1, and return value.) +r4-r8: Volatile. (Parameters 2-6.) +cr0: Volatile (cr0.SO is the return error condition) +cr1, cr5-7: Nonvolatile. +lr: Nonvolatile. + +All floating point and vector data registers as well as control and status +registers are nonvolatile. + +Invocation +---------- +The syscall is performed with the sc instruction, and returns with execution +continuing at the instruction following the sc instruction. + +Transactional Memory +-------------------- +Syscall behavior can change if the processor is in transactional or suspended +transaction state, and the syscall can affect the behavior of the transaction. + +If the processor is in suspended state when a syscall is made, the syscall +will be performed as normal, and will return as normal. The syscall will be +performed in suspended state, so its side effects will be persistent according +to the usual transactional memory semantics. A syscall may or may not result +in the transaction being doomed by hardware. + +If the processor is in transactional state when a syscall is made, then the +behavior depends on the presence of PPC_FEATURE2_HTM_NOSC in the AT_HWCAP2 ELF +auxiliary vector. + +- If present, which is the case for newer kernels, then the syscall will not + be performed and the transaction will be doomed by the kernel with the + failure code TM_CAUSE_SYSCALL | TM_CAUSE_PERSISTENT in the TEXASR SPR. + +- If not present (older kernels), then the kernel will suspend the + transactional state and the syscall will proceed as in the case of a + suspended state syscall, and will resume the transactional state before + returning to the caller. This case is not well defined or supported, so this + behavior should not be relied upon. + + +vsyscall +======== + +vsyscall calling sequence matches the syscall calling sequence, with the +following differences. Some vsyscalls may have different calling sequences. + +Parameters and return value +--------------------------- +r0 is not used as an input. The vsyscall is selected by its address. + +Stack +----- +The vsyscall may or may not use the caller's stack frame save areas. + +Register preservation rules +--------------------------- +r0: Volatile. +cr1, cr5-7: Volatile. +lr: Volatile. + +Invocation +---------- +The vsyscall is performed with a branch-with-link instruction to the vsyscall +function address. + +Transactional Memory +-------------------- +vsyscalls will run in the same transactional state as the caller. A vsyscall +may or may not result in the transaction being doomed by hardware. diff --git a/MAINTAINERS b/MAINTAINERS index fb4d381a6ecf..269bce8a4838 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3523,14 +3523,14 @@ F: drivers/net/ethernet/chelsio/cxgb4vf/ CXL (IBM Coherent Accelerator Processor Interface CAPI) DRIVER M: Ian Munsie <imunsie@au1.ibm.com> -M: Michael Neuling <mikey@neuling.org> +M: Frederic Barrat <fbarrat@linux.vnet.ibm.com> L: linuxppc-dev@lists.ozlabs.org S: Supported +F: arch/powerpc/platforms/powernv/pci-cxl.c F: drivers/misc/cxl/ F: include/misc/cxl* F: include/uapi/misc/cxl.h F: Documentation/powerpc/cxl.txt -F: Documentation/powerpc/cxl.txt F: Documentation/ABI/testing/sysfs-class-cxl CXLFLASH (IBM Coherent Accelerator Processor Interface CAPI Flash) SCSI DRIVER diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 927d2ab2ce08..569b4f9271a4 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -12,11 +12,6 @@ config 64BIT bool default y if PPC64 -config WORD_SIZE - int - default 64 if PPC64 - default 32 if !PPC64 - config ARCH_PHYS_ADDR_T_64BIT def_bool PPC64 || PHYS_64BIT @@ -101,7 +96,7 @@ config PPC select VIRT_TO_BUS if !PPC64 select HAVE_IDE select HAVE_IOREMAP_PROT - select HAVE_EFFICIENT_UNALIGNED_ACCESS if !CPU_LITTLE_ENDIAN + select HAVE_EFFICIENT_UNALIGNED_ACCESS if !(CPU_LITTLE_ENDIAN && POWER7_CPU) select HAVE_KPROBES select HAVE_ARCH_KGDB select HAVE_KRETPROBES @@ -167,6 +162,7 @@ config PPC select GENERIC_CPU_AUTOPROBE select HAVE_VIRT_CPU_ACCOUNTING select HAVE_ARCH_HARDENED_USERCOPY + select HAVE_KERNEL_GZIP config GENERIC_CSUM def_bool CPU_LITTLE_ENDIAN @@ -637,7 +633,7 @@ config FORCE_MAX_ZONEORDER int "Maximum zone order" range 8 9 if PPC64 && PPC_64K_PAGES default "9" if PPC64 && PPC_64K_PAGES - range 9 13 if PPC64 && !PPC_64K_PAGES + range 13 13 if PPC64 && !PPC_64K_PAGES default "13" if PPC64 && !PPC_64K_PAGES range 9 64 if PPC32 && PPC_16K_PAGES default "9" if PPC32 && PPC_16K_PAGES diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile index 1934707bf321..50d020ac0f48 100644 --- a/arch/powerpc/Makefile +++ b/arch/powerpc/Makefile @@ -43,31 +43,24 @@ NM := $(NM) --synthetic endif endif -ifeq ($(CONFIG_PPC64),y) -ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) -OLDARCH := ppc64le -else -OLDARCH := ppc64 -endif -else -ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) -OLDARCH := ppcle -else -OLDARCH := ppc -endif -endif +# BITS is used as extension for files which are available in a 32 bit +# and a 64 bit version to simplify shared Makefiles. +# e.g.: obj-y += foo_$(BITS).o +export BITS -# It seems there are times we use this Makefile without -# including the config file, but this replicates the old behaviour -ifeq ($(CONFIG_WORD_SIZE),) -CONFIG_WORD_SIZE := 32 +ifdef CONFIG_PPC64 + BITS := 64 +else + BITS := 32 endif -UTS_MACHINE := $(OLDARCH) +machine-y = ppc +machine-$(CONFIG_PPC64) += 64 +machine-$(CONFIG_CPU_LITTLE_ENDIAN) += le +UTS_MACHINE := $(subst $(space),,$(machine-y)) ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) override LD += -EL -override CROSS32AS += -mlittle-endian LDEMULATION := lppc GNUTARGET := powerpcle MULTIPLEWORD := -mno-multiple @@ -89,10 +82,10 @@ aflags-$(CONFIG_CPU_BIG_ENDIAN) += $(call cc-option,-mbig-endian) aflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mlittle-endian ifeq ($(HAS_BIARCH),y) -override AS += -a$(CONFIG_WORD_SIZE) -override LD += -m elf$(CONFIG_WORD_SIZE)$(LDEMULATION) -override CC += -m$(CONFIG_WORD_SIZE) -override AR := GNUTARGET=elf$(CONFIG_WORD_SIZE)-$(GNUTARGET) $(AR) +override AS += -a$(BITS) +override LD += -m elf$(BITS)$(LDEMULATION) +override CC += -m$(BITS) +override AR := GNUTARGET=elf$(BITS)-$(GNUTARGET) $(AR) endif LDFLAGS_vmlinux-y := -Bstatic @@ -179,7 +172,7 @@ KBUILD_CFLAGS += $(call cc-option,-msoft-float) KBUILD_CFLAGS += -pipe -Iarch/$(ARCH) $(CFLAGS-y) CPP = $(CC) -E $(KBUILD_CFLAGS) -CHECKFLAGS += -m$(CONFIG_WORD_SIZE) -D__powerpc__ -D__powerpc$(CONFIG_WORD_SIZE)__ +CHECKFLAGS += -m$(BITS) -D__powerpc__ -D__powerpc$(BITS)__ ifdef CONFIG_CPU_BIG_ENDIAN CHECKFLAGS += -D__BIG_ENDIAN__ else @@ -234,7 +227,7 @@ KBUILD_CFLAGS += $(cpu-as-y) KBUILD_AFLAGS += $(aflags-y) KBUILD_CFLAGS += $(cflags-y) -head-y := arch/powerpc/kernel/head_$(CONFIG_WORD_SIZE).o +head-y := arch/powerpc/kernel/head_$(BITS).o head-$(CONFIG_8xx) := arch/powerpc/kernel/head_8xx.o head-$(CONFIG_40x) := arch/powerpc/kernel/head_40x.o head-$(CONFIG_44x) := arch/powerpc/kernel/head_44x.o diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile index 1a2a6e8dc40d..eae2dc8bc218 100644 --- a/arch/powerpc/boot/Makefile +++ b/arch/powerpc/boot/Makefile @@ -19,10 +19,15 @@ all: $(obj)/zImage +compress-$(CONFIG_KERNEL_GZIP) := CONFIG_KERNEL_GZIP +compress-$(CONFIG_KERNEL_XZ) := CONFIG_KERNEL_XZ + BOOTCFLAGS := -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \ -fno-strict-aliasing -Os -msoft-float -pipe \ -fomit-frame-pointer -fno-builtin -fPIC -nostdinc \ - -isystem $(shell $(CROSS32CC) -print-file-name=include) + -isystem $(shell $(CROSS32CC) -print-file-name=include) \ + -D$(compress-y) + ifdef CONFIG_PPC64_BOOT_WRAPPER BOOTCFLAGS += -m64 endif @@ -59,13 +64,30 @@ $(obj)/treeboot-currituck.o: BOOTCFLAGS += -mcpu=405 $(obj)/treeboot-akebono.o: BOOTCFLAGS += -mcpu=405 $(obj)/virtex405-head.o: BOOTAFLAGS += -mcpu=405 +# The pre-boot decompressors pull in a lot of kernel headers and other source +# files. This creates a bit of a dependency headache since we need to copy +# these files into the build dir, fix up any includes and ensure that dependent +# files are copied in the right order. + +# these need to be seperate variables because they are copied out of different +# directories in the kernel tree. Sure you COULd merge them, but it's a +# cure-is-worse-than-disease situation. +zlib-decomp-$(CONFIG_KERNEL_GZIP) := decompress_inflate.c +zlib-$(CONFIG_KERNEL_GZIP) := inffast.c inflate.c inftrees.c +zlibheader-$(CONFIG_KERNEL_GZIP) := inffast.h inffixed.h inflate.h inftrees.h infutil.h +zliblinuxheader-$(CONFIG_KERNEL_GZIP) := zlib.h zconf.h zutil.h -zlib := inffast.c inflate.c inftrees.c -zlibheader := inffast.h inffixed.h inflate.h inftrees.h infutil.h -zliblinuxheader := zlib.h zconf.h zutil.h +$(addprefix $(obj)/, decompress.o): \ + $(addprefix $(obj)/,$(zlib-decomp-y)) -$(addprefix $(obj)/,$(zlib) cuboot-c2k.o gunzip_util.o main.o): \ - $(addprefix $(obj)/,$(zliblinuxheader)) $(addprefix $(obj)/,$(zlibheader)) +$(addprefix $(obj)/, $(zlib-decomp-y)): \ + $(addprefix $(obj)/,$(zliblinuxheader-y)) \ + $(addprefix $(obj)/,$(zlibheader-y)) \ + $(addprefix $(obj)/,$(zlib-y)) + +$(addprefix $(obj)/,$(zlib-y)): \ + $(addprefix $(obj)/,$(zliblinuxheader-y)) \ + $(addprefix $(obj)/,$(zlibheader-y)) libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c libfdtheader := fdt.h libfdt.h libfdt_internal.h @@ -73,10 +95,10 @@ libfdtheader := fdt.h libfdt.h libfdt_internal.h $(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o): \ $(addprefix $(obj)/,$(libfdtheader)) -src-wlib-y := string.S crt0.S crtsavres.S stdio.c main.c \ +src-wlib-y := string.S crt0.S crtsavres.S stdio.c decompress.c main.c \ $(libfdt) libfdt-wrapper.c \ ns16550.c serial.c simple_alloc.c div64.S util.S \ - gunzip_util.c elf_util.c $(zlib) devtree.c stdlib.c \ + elf_util.c $(zlib-y) devtree.c stdlib.c \ oflib.c ofconsole.c cuboot.c mpsc.c cpm-serial.c \ uartlite.c mpc52xx-psc.c opal.c opal-calls.S src-wlib-$(CONFIG_40x) += 4xx.c planetcore.c @@ -125,23 +147,20 @@ obj-wlib := $(addsuffix .o, $(basename $(addprefix $(obj)/, $(src-wlib)))) obj-plat := $(addsuffix .o, $(basename $(addprefix $(obj)/, $(src-plat)))) obj-plat: $(libfdt) -quiet_cmd_copy_zlib = COPY $@ - cmd_copy_zlib = sed "s@__used@@;s@<linux/\([^>]*\).*@\"\1\"@" $< > $@ +quiet_cmd_copy_kern_src = COPY $@ + cmd_copy_kern_src = sed -f $(srctree)/arch/powerpc/boot/fixup-headers.sed $< > $@ -quiet_cmd_copy_zlibheader = COPY $@ - cmd_copy_zlibheader = sed "s@<linux/\([^>]*\).*@\"\1\"@" $< > $@ -# stddef.h for NULL -quiet_cmd_copy_zliblinuxheader = COPY $@ - cmd_copy_zliblinuxheader = sed "s@<linux/string.h>@\"string.h\"@;s@<linux/kernel.h>@<stddef.h>@;s@<linux/\([^>]*\).*@\"\1\"@" $< > $@ +$(addprefix $(obj)/,$(zlib-y)): $(obj)/%: $(srctree)/lib/zlib_inflate/% + $(call cmd,copy_kern_src) -$(addprefix $(obj)/,$(zlib)): $(obj)/%: $(srctree)/lib/zlib_inflate/% - $(call cmd,copy_zlib) +$(addprefix $(obj)/,$(zlibheader-y)): $(obj)/%: $(srctree)/lib/zlib_inflate/% + $(call cmd,copy_kern_src) -$(addprefix $(obj)/,$(zlibheader)): $(obj)/%: $(srctree)/lib/zlib_inflate/% - $(call cmd,copy_zlibheader) +$(addprefix $(obj)/,$(zliblinuxheader-y)): $(obj)/%: $(srctree)/include/linux/% + $(call cmd,copy_kern_src) -$(addprefix $(obj)/,$(zliblinuxheader)): $(obj)/%: $(srctree)/include/linux/% - $(call cmd,copy_zliblinuxheader) +$(addprefix $(obj)/,$(zlib-decomp-y)): $(obj)/%: $(srctree)/lib/% + $(call cmd,copy_kern_src) quiet_cmd_copy_libfdt = COPY $@ cmd_copy_libfdt = cp $< $@ @@ -150,17 +169,17 @@ $(addprefix $(obj)/,$(libfdt) $(libfdtheader)): $(obj)/%: $(srctree)/scripts/dtc $(call cmd,copy_libfdt) $(obj)/empty.c: - @touch $@ + $(Q)touch $@ $(obj)/zImage.lds: $(obj)/%: $(srctree)/$(src)/%.S $(CROSS32CC) $(cpp_flags) -E -Wp,-MD,$(depfile) -P -Upowerpc \ -D__ASSEMBLY__ -DLINKER_SCRIPT -o $@ $< $(obj)/zImage.coff.lds $(obj)/zImage.ps3.lds : $(obj)/%: $(srctree)/$(src)/%.S - @cp $< $@ + $(Q)cp $< $@ -clean-files := $(zlib) $(zlibheader) $(zliblinuxheader) \ - $(libfdt) $(libfdtheader) \ +clean-files := $(zlib-) $(zlibheader-) $(zliblinuxheader-) \ + $(zlib-decomp-) $(libfdt) $(libfdtheader) \ empty.c zImage.coff.lds zImage.ps3.lds zImage.lds quiet_cmd_bootcc = BOOTCC $@ @@ -207,10 +226,14 @@ CROSSWRAP := -C "$(CROSS_COMPILE)" endif endif +compressor-$(CONFIG_KERNEL_GZIP) := gz +compressor-$(CONFIG_KERNEL_XZ) := xz + # args (to if_changed): 1 = (this rule), 2 = platform, 3 = dts 4=dtb 5=initrd quiet_cmd_wrap = WRAP $@ - cmd_wrap =$(CONFIG_SHELL) $(wrapper) -c -o $@ -p $2 $(CROSSWRAP) \ - $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) vmlinux + cmd_wrap =$(CONFIG_SHELL) $(wrapper) -Z $(compressor-y) -c -o $@ -p $2 \ + $(CROSSWRAP) $(if $3, -s $3)$(if $4, -d $4)$(if $5, -i $5) \ + vmlinux image-$(CONFIG_PPC_PSERIES) += zImage.pseries image-$(CONFIG_PPC_POWERNV) += zImage.pseries @@ -391,9 +414,9 @@ image-y := vmlinux.strip endif $(obj)/zImage: $(addprefix $(obj)/, $(image-y)) - @rm -f $@; ln $< $@ + $(Q)rm -f $@; ln $< $@ $(obj)/zImage.initrd: $(addprefix $(obj)/, $(initrd-y)) - @rm -f $@; ln $< $@ + $(Q)rm -f $@; ln $< $@ # Only install the vmlinux install: $(CONFIGURE) $(addprefix $(obj)/, $(image-y)) @@ -410,8 +433,9 @@ clean-files += $(image-) $(initrd-) cuImage.* dtbImage.* treeImage.* \ zImage.maple simpleImage.* otheros.bld *.dtb # clean up files cached by wrapper -clean-kernel := vmlinux.strip vmlinux.bin -clean-kernel += $(addsuffix .gz,$(clean-kernel)) +clean-kernel-base := vmlinux.strip vmlinux.bin +clean-kernel := $(addsuffix .gz,$(clean-kernel-base)) +clean-kernel += $(addsuffix .xz,$(clean-kernel-base)) # If not absolute clean-files are relative to $(obj). clean-files += $(addprefix $(objtree)/, $(clean-kernel)) diff --git a/arch/powerpc/boot/cuboot-c2k.c b/arch/powerpc/boot/cuboot-c2k.c index e43594950ba3..9309c51f1d65 100644 --- a/arch/powerpc/boot/cuboot-c2k.c +++ b/arch/powerpc/boot/cuboot-c2k.c @@ -18,7 +18,6 @@ #include "io.h" #include "ops.h" #include "elf.h" -#include "gunzip_util.h" #include "mv64x60.h" #include "cuboot.h" #include "ppcboot.h" diff --git a/arch/powerpc/boot/decompress.c b/arch/powerpc/boot/decompress.c new file mode 100644 index 000000000000..3aff4423ad01 --- /dev/null +++ b/arch/powerpc/boot/decompress.c @@ -0,0 +1,148 @@ +/* + * Wrapper around the kernel's pre-boot decompression library. + * + * Copyright (C) IBM Corporation 2016. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "elf.h" +#include "page.h" +#include "string.h" +#include "stdio.h" +#include "ops.h" +#include "reg.h" +#include "types.h" + +/* + * The decompressor_*.c files play #ifdef games so they can be used in both + * pre-boot and regular kernel code. We need these definitions to make the + * includes work. + */ + +#define STATIC static +#define INIT +#define __always_inline inline + +/* + * The build process will copy the required zlib source files and headers + * out of lib/ and "fix" the includes so they do not pull in other kernel + * headers. + */ + +#ifdef CONFIG_KERNEL_GZIP +# include "decompress_inflate.c" +#endif + +#ifdef CONFIG_KERNEL_XZ +# include "xz_config.h" +# include "../../../lib/decompress_unxz.c" +#endif + +/* globals for tracking the state of the decompression */ +static unsigned long decompressed_bytes; +static unsigned long limit; +static unsigned long skip; +static char *output_buffer; + +/* + * flush() is called by __decompress() when the decompressor's scratch buffer is + * full. + */ +static long flush(void *v, unsigned long buffer_size) +{ + unsigned long end = decompressed_bytes + buffer_size; + unsigned long size = buffer_size; + unsigned long offset = 0; + char *in = v; + char *out; + + /* + * if we hit our decompression limit, we need to fake an error to abort + * the in-progress decompression. + */ + if (decompressed_bytes >= limit) + return -1; + + /* skip this entire block */ + if (end <= skip) { + decompressed_bytes += buffer_size; + return buffer_size; + } + + /* skip some data at the start, but keep the rest of the block */ + if (decompressed_bytes < skip && end > skip) { + offset = skip - decompressed_bytes; + + in += offset; + size -= offset; + decompressed_bytes += offset; + } + + out = &output_buffer[decompressed_bytes - skip]; + size = min(decompressed_bytes + size, limit) - decompressed_bytes; + + memcpy(out, in, size); + decompressed_bytes += size; + + return buffer_size; +} + +static void print_err(char *s) +{ + /* suppress the "error" when we terminate the decompressor */ + if (decompressed_bytes >= limit) + return; + + printf("Decompression error: '%s'\n\r", s); +} + +/** + * partial_decompress - decompresses part or all of a compressed buffer + * @inbuf: input buffer + * @input_size: length of the input buffer + * @outbuf: input buffer + * @output_size: length of the input buffer + * @skip number of output bytes to ignore + * + * This function takes compressed data from inbuf, decompresses and write it to + * outbuf. Once output_size bytes are written to the output buffer, or the + * stream is exhausted the function will return the number of bytes that were + * decompressed. Otherwise it will return whatever error code the decompressor + * reported (NB: This is specific to each decompressor type). + * + * The skip functionality is mainly there so the program and discover + * the size of the compressed image so that it can ask firmware (if present) + * for an appropriately sized buffer. + */ +long partial_decompress(void *inbuf, unsigned long input_size, + void *outbuf, unsigned long output_size, unsigned long _skip) +{ + int ret; + + /* + * The skipped bytes needs to be included in the size of data we want + * to decompress. + */ + output_size += _skip; + + decompressed_bytes = 0; + output_buffer = outbuf; + limit = output_size; + skip = _skip; + + ret = __decompress(inbuf, input_size, NULL, flush, outbuf, + output_size, NULL, print_err); + + /* + * If decompression was aborted due to an actual error rather than + * a fake error that we used to abort, then we should report it. + */ + if (decompressed_bytes < limit) + return ret; + + return decompressed_bytes - skip; +} diff --git a/arch/powerpc/boot/fixup-headers.sed b/arch/powerpc/boot/fixup-headers.sed new file mode 100644 index 000000000000..96362428eb37 --- /dev/null +++ b/arch/powerpc/boot/fixup-headers.sed @@ -0,0 +1,12 @@ +# Copyright 2016 IBM Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License version 2 or later as +# published by the Free Software Foundation. + +s@#include <linux/decompress/mm\.h>@@; +s@\"zlib_inflate/\([^\"]*\).*@"\1"@; +s@<linux/kernel.h>@<stddef.h>@; + +s@__used@@; +s@<linux/\([^>]*\).*@"\1"@; diff --git a/arch/powerpc/boot/gunzip_util.c b/arch/powerpc/boot/gunzip_util.c deleted file mode 100644 index 9dc52501de83..000000000000 --- a/arch/powerpc/boot/gunzip_util.c +++ /dev/null @@ -1,204 +0,0 @@ -/* - * Copyright 2007 David Gibson, IBM Corporation. - * Based on earlier work, Copyright (C) Paul Mackerras 1997. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - */ - -#include <stddef.h> -#include "string.h" -#include "stdio.h" -#include "ops.h" -#include "gunzip_util.h" - -#define HEAD_CRC 2 -#define EXTRA_FIELD 4 -#define ORIG_NAME 8 -#define COMMENT 0x10 -#define RESERVED 0xe0 - -/** - * gunzip_start - prepare to decompress gzip data - * @state: decompressor state structure to be initialized - * @src: buffer containing gzip compressed or uncompressed data - * @srclen: size in bytes of the buffer at src - * - * If the buffer at @src contains a gzip header, this function - * initializes zlib to decompress the data, storing the decompression - * state in @state. The other functions in this file can then be used - * to decompress data from the gzipped stream. - * - * If the buffer at @src does not contain a gzip header, it is assumed - * to contain uncompressed data. The buffer information is recorded - * in @state and the other functions in this file will simply copy - * data from the uncompressed data stream at @src. - * - * Any errors, such as bad compressed data, cause an error to be - * printed an the platform's exit() function to be called. - */ -void gunzip_start(struct gunzip_state *state, void *src, int srclen) -{ - char *hdr = src; - int hdrlen = 0; - - memset(state, 0, sizeof(*state)); - - /* Check for gzip magic number */ - if ((hdr[0] == 0x1f) && (hdr[1] == 0x8b)) { - /* gzip data, initialize zlib parameters */ - int r, flags; - - state->s.workspace = state->scratch; - if (zlib_inflate_workspacesize() > sizeof(state->scratch)) - fatal("insufficient scratch space for gunzip\n\r"); - - /* skip header */ - hdrlen = 10; - flags = hdr[3]; - if (hdr[2] != Z_DEFLATED || (flags & RESERVED) != 0) - fatal("bad gzipped data\n\r"); - if ((flags & EXTRA_FIELD) != 0) - hdrlen = 12 + hdr[10] + (hdr[11] << 8); - if ((flags & ORIG_NAME) != 0) - while (hdr[hdrlen++] != 0) - ; - if ((flags & COMMENT) != 0) - while (hdr[hdrlen++] != 0) - ; - if ((flags & HEAD_CRC) != 0) - hdrlen += 2; - if (hdrlen >= srclen) - fatal("gunzip_start: ran out of data in header\n\r"); - - r = zlib_inflateInit2(&state->s, -MAX_WBITS); - if (r != Z_OK) - fatal("inflateInit2 returned %d\n\r", r); - } - - state->s.total_in = hdrlen; - state->s.next_in = src + hdrlen; - state->s.avail_in = srclen - hdrlen; -} - -/** - * gunzip_partial - extract bytes from a gzip data stream - * @state: gzip state structure previously initialized by gunzip_start() - * @dst: buffer to store extracted data - * @dstlen: maximum number of bytes to extract - * - * This function extracts at most @dstlen bytes from the data stream - * previously associated with @state by gunzip_start(), decompressing - * if necessary. Exactly @dstlen bytes are extracted unless the data - * stream doesn't contain enough bytes, in which case the entire - * remainder of the stream is decompressed. - * - * Returns the actual number of bytes extracted. If any errors occur, - * such as a corrupted compressed stream, an error is printed an the - * platform's exit() function is called. - */ -int gunzip_partial(struct gunzip_state *state, void *dst, int dstlen) -{ - int len; - - if (state->s.workspace) { - /* gunzipping */ - int r; - - state->s.next_out = dst; - state->s.avail_out = dstlen; - r = zlib_inflate(&state->s, Z_FULL_FLUSH); - if (r != Z_OK && r != Z_STREAM_END) - fatal("inflate returned %d msg: %s\n\r", r, state->s.msg); - len = state->s.next_out - (Byte *)dst; - } else { - /* uncompressed image */ - len = min(state->s.avail_in, (uLong)dstlen); - memcpy(dst, state->s.next_in, len); - state->s.next_in += len; - state->s.avail_in -= len; - } - return len; -} - -/** - * gunzip_exactly - extract a fixed number of bytes from a gzip data stream - * @state: gzip state structure previously initialized by gunzip_start() - * @dst: buffer to store extracted data - * @dstlen: number of bytes to extract - * - * This function extracts exactly @dstlen bytes from the data stream - * previously associated with @state by gunzip_start(), decompressing - * if necessary. - * - * If there are less @dstlen bytes available in the data stream, or if - * any other errors occur, such as a corrupted compressed stream, an - * error is printed an the platform's exit() function is called. - */ -void gunzip_exactly(struct gunzip_state *state, void *dst, int dstlen) -{ - int len; - - len = gunzip_partial(state, dst, dstlen); - if (len < dstlen) - fatal("\n\rgunzip_exactly: ran out of data!" - " Wanted %d, got %d.\n\r", dstlen, len); -} - -/** - * gunzip_discard - discard bytes from a gzip data stream - * @state: gzip state structure previously initialized by gunzip_start() - * @len: number of bytes to discard - * - * This function extracts, then discards exactly @len bytes from the - * data stream previously associated with @state by gunzip_start(). - * Subsequent gunzip_partial(), gunzip_exactly() or gunzip_finish() - * calls will extract the data following the discarded bytes in the - * data stream. - * - * If there are less @len bytes available in the data stream, or if - * any other errors occur, such as a corrupted compressed stream, an - * error is printed an the platform's exit() function is called. - */ -void gunzip_discard(struct gunzip_state *state, int len) -{ - static char discard_buf[128]; - - while (len > sizeof(discard_buf)) { - gunzip_exactly(state, discard_buf, sizeof(discard_buf)); - len -= sizeof(discard_buf); - } - - if (len > 0) - gunzip_exactly(state, discard_buf, len); -} - -/** - * gunzip_finish - extract all remaining bytes from a gzip data stream - * @state: gzip state structure previously initialized by gunzip_start() - * @dst: buffer to store extracted data - * @dstlen: maximum number of bytes to extract - * - * This function extracts all remaining data, or at most @dstlen - * bytes, from the stream previously associated with @state by - * gunzip_start(). zlib is then shut down, so it is an error to use - * any of the functions in this file on @state until it is - * re-initialized with another call to gunzip_start(). - * - * If any errors occur, such as a corrupted compressed stream, an - * error is printed an the platform's exit() function is called. - */ -int gunzip_finish(struct gunzip_state *state, void *dst, int dstlen) -{ - int len; - - len = gunzip_partial(state, dst, dstlen); - - if (state->s.workspace) { - zlib_inflateEnd(&state->s); - } - - return len; -} diff --git a/arch/powerpc/boot/gunzip_util.h b/arch/powerpc/boot/gunzip_util.h deleted file mode 100644 index b3dfa6e87b3a..000000000000 --- a/arch/powerpc/boot/gunzip_util.h +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Decompression convenience functions - * - * Copyright 2007 David Gibson, IBM Corporation. - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - */ -#ifndef _PPC_BOOT_GUNZIP_UTIL_H_ -#define _PPC_BOOT_GUNZIP_UTIL_H_ - -#include "zlib.h" - -/* - * These functions are designed to make life easy for decompressing - * kernel images, initrd images or any other gzip compressed image, - * particularly if its useful to decompress part of the image (e.g. to - * examine headers) before decompressing the remainder. - * - * To use: - * - declare a gunzip_state structure - * - use gunzip_start() to initialize the state, associating it - * with a stream of compressed data - * - use gunzip_partial(), gunzip_exactly() and gunzip_discard() - * in any combination to extract pieces of data from the stream - * - Finally use gunzip_finish() to extract the tail of the - * compressed stream and wind up zlib - */ - -/* scratch space for gunzip; 46912 is from zlib_inflate_workspacesize() */ -#define GUNZIP_SCRATCH_SIZE 46912 - -struct gunzip_state { - z_stream s; - char scratch[46912]; -}; - -void gunzip_start(struct gunzip_state *state, void *src, int srclen); -int gunzip_partial(struct gunzip_state *state, void *dst, int dstlen); -void gunzip_exactly(struct gunzip_state *state, void *dst, int len); -void gunzip_discard(struct gunzip_state *state, int len); -int gunzip_finish(struct gunzip_state *state, void *dst, int len); - -#endif /* _PPC_BOOT_GUNZIP_UTIL_H_ */ diff --git a/arch/powerpc/boot/main.c b/arch/powerpc/boot/main.c index d80161b633f4..f7a184b6c35b 100644 --- a/arch/powerpc/boot/main.c +++ b/arch/powerpc/boot/main.c @@ -15,11 +15,8 @@ #include "string.h" #include "stdio.h" #include "ops.h" -#include "gunzip_util.h" #include "reg.h" -static struct gunzip_state gzstate; - struct addr_range { void *addr; unsigned long size; @@ -30,15 +27,14 @@ struct addr_range { static struct addr_range prep_kernel(void) { char elfheader[256]; - void *vmlinuz_addr = _vmlinux_start; + unsigned char *vmlinuz_addr = (unsigned char *)_vmlinux_start; unsigned long vmlinuz_size = _vmlinux_end - _vmlinux_start; void *addr = 0; struct elf_info ei; - int len; + long len; - /* gunzip the ELF header of the kernel */ - gunzip_start(&gzstate, vmlinuz_addr, vmlinuz_size); - gunzip_exactly(&gzstate, elfheader, sizeof(elfheader)); + partial_decompress(vmlinuz_addr, vmlinuz_size, + elfheader, sizeof(elfheader), 0); if (!parse_elf64(elfheader, &ei) && !parse_elf32(elfheader, &ei)) fatal("Error: not a valid PPC32 or PPC64 ELF file!\n\r"); @@ -51,7 +47,7 @@ static struct addr_range prep_kernel(void) * the kernel bss must be claimed (it will be zero'd by the * kernel itself) */ - printf("Allocating 0x%lx bytes for kernel ...\n\r", ei.memsize); + printf("Allocating 0x%lx bytes for kernel...\n\r", ei.memsize); if (platform_ops.vmlinux_alloc) { addr = platform_ops.vmlinux_alloc(ei.memsize); @@ -71,16 +67,21 @@ static struct addr_range prep_kernel(void) "device tree\n\r"); } - /* Finally, gunzip the kernel */ - printf("gunzipping (0x%p <- 0x%p:0x%p)...", addr, + /* Finally, decompress the kernel */ + printf("Decompressing (0x%p <- 0x%p:0x%p)...\n\r", addr, vmlinuz_addr, vmlinuz_addr+vmlinuz_size); - /* discard up to the actual load data */ - gunzip_discard(&gzstate, ei.elfoffset - sizeof(elfheader)); - len = gunzip_finish(&gzstate, addr, ei.loadsize); + + len = partial_decompress(vmlinuz_addr, vmlinuz_size, + addr, ei.loadsize, ei.elfoffset); + + if (len < 0) + fatal("Decompression failed with error code %ld\n\r", len); + if (len != ei.loadsize) - fatal("ran out of data! only got 0x%x of 0x%lx bytes.\n\r", - len, ei.loadsize); - printf("done 0x%x bytes\n\r", len); + fatal("Decompression error: got 0x%lx bytes, expected 0x%lx.\n\r", + len, ei.loadsize); + + printf("Done! Decompressed 0x%lx bytes\n\r", len); flush_cache(addr, ei.loadsize); diff --git a/arch/powerpc/boot/ops.h b/arch/powerpc/boot/ops.h index e19b64ef977a..309d1b127e96 100644 --- a/arch/powerpc/boot/ops.h +++ b/arch/powerpc/boot/ops.h @@ -260,4 +260,7 @@ int __ilog2_u32(u32 n) return 31 - bit; } +long partial_decompress(void *inbuf, unsigned long input_size, void *outbuf, + unsigned long output_size, unsigned long skip); + #endif /* _PPC_BOOT_OPS_H_ */ diff --git a/arch/powerpc/boot/stdbool.h b/arch/powerpc/boot/stdbool.h new file mode 100644 index 000000000000..f818efb08891 --- /dev/null +++ b/arch/powerpc/boot/stdbool.h @@ -0,0 +1,14 @@ +/* + * Copyright (C) IBM Corporation 2016. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * This file is only necessary because some of the pre-boot decompressors + * expect stdbool.h to be available. + * + */ + +#include "types.h" diff --git a/arch/powerpc/boot/stdint.h b/arch/powerpc/boot/stdint.h new file mode 100644 index 000000000000..c1c853be7490 --- /dev/null +++ b/arch/powerpc/boot/stdint.h @@ -0,0 +1,13 @@ +/* + * Copyright (C) IBM Corporation 2016. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * This file is only necessary because some of the pre-boot decompressors + * expect stdint.h to be available. + */ + +#include "types.h" diff --git a/arch/powerpc/boot/types.h b/arch/powerpc/boot/types.h index 85565a89bcc2..af6b66b842c4 100644 --- a/arch/powerpc/boot/types.h +++ b/arch/powerpc/boot/types.h @@ -1,6 +1,8 @@ #ifndef _TYPES_H_ #define _TYPES_H_ +#include <stdbool.h> + #define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) typedef unsigned char u8; @@ -34,4 +36,16 @@ typedef s64 int64_t; (void) (&_x == &_y); \ _x > _y ? _x : _y; }) +#define min_t(type, a, b) min(((type) a), ((type) b)) +#define max_t(type, a, b) max(((type) a), ((type) b)) + +typedef int bool; + +#ifndef true +#define true 1 +#endif + +#ifndef false +#define false 0 +#endif #endif /* _TYPES_H_ */ diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper index 6681ec3625c9..404b3aabdb4d 100755 --- a/arch/powerpc/boot/wrapper +++ b/arch/powerpc/boot/wrapper @@ -20,6 +20,8 @@ # -D dir specify directory containing data files used by script # (default ./arch/powerpc/boot) # -W dir specify working directory for temporary files (default .) +# -z use gzip (legacy) +# -Z zsuffix compression to use (gz, xz or none) # Stop execution if any command fails set -e @@ -38,7 +40,7 @@ dtb= dts= cacheit= binary= -gzip=.gz +compression=.gz pie= format= @@ -59,7 +61,8 @@ tmpdir=. usage() { echo 'Usage: wrapper [-o output] [-p platform] [-i initrd]' >&2 echo ' [-d devtree] [-s tree.dts] [-c] [-C cross-prefix]' >&2 - echo ' [-D datadir] [-W workingdir] [--no-gzip] [vmlinux]' >&2 + echo ' [-D datadir] [-W workingdir] [-Z (gz|xz|none)]' >&2 + echo ' [--no-compression] [vmlinux]' >&2 exit 1 } @@ -126,8 +129,24 @@ while [ "$#" -gt 0 ]; do [ "$#" -gt 0 ] || usage tmpdir="$1" ;; + -z) + compression=.gz + ;; + -Z) + shift + [ "$#" -gt 0 ] || usage + [ "$1" != "gz" -o "$1" != "xz" -o "$1" != "none" ] || usage + + compression=".$1" + + if [ $compression = ".none" ]; then + compression= + fi + ;; --no-gzip) - gzip= + # a "feature" of the the wrapper script is that it can be used outside + # the kernel tree. So keeping this around for backwards compatibility. + compression= ;; -?) usage @@ -140,6 +159,7 @@ while [ "$#" -gt 0 ]; do shift done + if [ -n "$dts" ]; then if [ ! -r "$dts" -a -r "$object/dts/$dts" ]; then dts="$object/dts/$dts" @@ -212,7 +232,7 @@ miboot|uboot*) ;; cuboot*) binary=y - gzip= + compression= case "$platform" in *-mpc866ads|*-mpc885ads|*-adder875*|*-ep88xc) platformo=$object/cuboot-8xx.o @@ -243,7 +263,7 @@ cuboot*) ps3) platformo="$object/ps3-head.o $object/ps3-hvcall.o $object/ps3.o" lds=$object/zImage.ps3.lds - gzip= + compression= ext=bin objflags="-O binary --set-section-flags=.bss=contents,alloc,load,data" ksection=.kernel:vmlinux.bin @@ -310,27 +330,37 @@ mvme7100) esac vmz="$tmpdir/`basename \"$kernel\"`.$ext" -if [ -z "$cacheit" -o ! -f "$vmz$gzip" -o "$vmz$gzip" -ot "$kernel" ]; then - ${CROSS}objcopy $objflags "$kernel" "$vmz.$$" - strip_size=$(stat -c %s $vmz.$$) +# Calculate the vmlinux.strip size +${CROSS}objcopy $objflags "$kernel" "$vmz.$$" +strip_size=$(stat -c %s $vmz.$$) - if [ -n "$gzip" ]; then +if [ -z "$cacheit" -o ! -f "$vmz$compression" -o "$vmz$compression" -ot "$kernel" ]; then + # recompress the image if we need to + case $compression in + .xz) + xz --check=crc32 -f -6 "$vmz.$$" + ;; + .gz) gzip -n -f -9 "$vmz.$$" - fi + ;; + *) + # drop the compression suffix so the stripped vmlinux is used + compression= + ;; + esac if [ -n "$cacheit" ]; then - mv -f "$vmz.$$$gzip" "$vmz$gzip" + mv -f "$vmz.$$$compression" "$vmz$compression" else vmz="$vmz.$$" fi else - # Calculate the vmlinux.strip size - ${CROSS}objcopy $objflags "$kernel" "$vmz.$$" - strip_size=$(stat -c %s $vmz.$$) rm -f $vmz.$$ fi +vmz="$vmz$compression" + if [ "$make_space" = "y" ]; then # Round the size to next higher MB limit round_size=$(((strip_size + 0xfffff) & 0xfff00000)) @@ -346,8 +376,6 @@ if [ "$make_space" = "y" ]; then fi fi -vmz="$vmz$gzip" - # Extract kernel version information, some platforms want to include # it in the image header version=`${CROSS}strings "$kernel" | grep '^Linux version [-0-9.]' | \ @@ -417,6 +445,7 @@ if [ "$platform" != "miboot" ]; then if [ -n "$link_address" ] ; then text_start="-Ttext $link_address" fi +#link everything ${CROSS}ld -m $format -T $lds $text_start $pie -o "$ofile" \ $platformo $tmp $object/wrapper.a rm $tmp diff --git a/arch/powerpc/boot/xz_config.h b/arch/powerpc/boot/xz_config.h new file mode 100644 index 000000000000..5c6afdbca642 --- /dev/null +++ b/arch/powerpc/boot/xz_config.h @@ -0,0 +1,39 @@ +#ifndef __XZ_CONFIG_H__ +#define __XZ_CONFIG_H__ + +/* + * most of this is copied from lib/xz/xz_private.h, we can't use their defines + * since the boot wrapper is not built in the same environment as the rest of + * the kernel. + */ + +#include "types.h" +#include "swab.h" + +static inline uint32_t swab32p(void *p) +{ + uint32_t *q = p; + + return swab32(*q); +} + +#ifdef __LITTLE_ENDIAN__ +#define get_le32(p) (*((uint32_t *) (p))) +#else +#define get_le32(p) swab32p(p) +#endif + +#define memeq(a, b, size) (memcmp(a, b, size) == 0) +#define memzero(buf, size) memset(buf, 0, size) + +/* prevent the inclusion of the xz-preboot MM headers */ +#define DECOMPR_MM_H +#define memmove memmove +#define XZ_EXTERN static + +/* xz.h needs to be included directly since we need enum xz_mode */ +#include "../../../include/linux/xz.h" + +#undef XZ_EXTERN + +#endif diff --git a/arch/powerpc/configs/powernv_defconfig b/arch/powerpc/configs/powernv_defconfig index dce352e9153b..d98b6eb3254f 100644 --- a/arch/powerpc/configs/powernv_defconfig +++ b/arch/powerpc/configs/powernv_defconfig @@ -15,6 +15,8 @@ CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y +CONFIG_LOG_BUF_SHIFT=18 +CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 CONFIG_NUMA_BALANCING=y CONFIG_CGROUPS=y CONFIG_MEMCG=y @@ -95,7 +97,7 @@ CONFIG_BLK_DEV_IDECD=y CONFIG_BLK_DEV_GENERIC=y CONFIG_BLK_DEV_AMD74XX=y CONFIG_BLK_DEV_SD=y -CONFIG_CHR_DEV_ST=y +CONFIG_CHR_DEV_ST=m CONFIG_BLK_DEV_SR=y CONFIG_BLK_DEV_SR_VENDOR=y CONFIG_CHR_DEV_SG=y @@ -107,7 +109,7 @@ CONFIG_SCSI_CXGB4_ISCSI=m CONFIG_SCSI_BNX2_ISCSI=m CONFIG_BE2ISCSI=m CONFIG_SCSI_MPT2SAS=m -CONFIG_SCSI_SYM53C8XX_2=y +CONFIG_SCSI_SYM53C8XX_2=m CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0 CONFIG_SCSI_IPR=y CONFIG_SCSI_QLA_FC=m @@ -149,10 +151,10 @@ CONFIG_TUN=m CONFIG_VETH=m CONFIG_VIRTIO_NET=m CONFIG_VHOST_NET=m -CONFIG_VORTEX=y +CONFIG_VORTEX=m CONFIG_ACENIC=m CONFIG_ACENIC_OMIT_TIGON_I=y -CONFIG_PCNET32=y +CONFIG_PCNET32=m CONFIG_TIGON3=y CONFIG_BNX2X=m CONFIG_CHELSIO_T1=m @@ -163,6 +165,7 @@ CONFIG_E1000=y CONFIG_E1000E=y CONFIG_IXGB=m CONFIG_IXGBE=m +CONFIG_I40E=m CONFIG_MLX4_EN=m CONFIG_MYRI10GE=m CONFIG_QLGE=m @@ -238,7 +241,7 @@ CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT4_FS=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y -CONFIG_REISERFS_FS=y +CONFIG_REISERFS_FS=m CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y @@ -253,10 +256,10 @@ CONFIG_NILFS2_FS=m CONFIG_AUTOFS4_FS=m CONFIG_FUSE_FS=m CONFIG_OVERLAY_FS=m -CONFIG_ISO9660_FS=y +CONFIG_ISO9660_FS=m CONFIG_UDF_FS=m CONFIG_MSDOS_FS=y -CONFIG_VFAT_FS=y +CONFIG_VFAT_FS=m CONFIG_PROC_KCORE=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y @@ -310,6 +313,8 @@ CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m CONFIG_CRYPTO_DEV_NX=y +CONFIG_CRYPTO_DEV_VMX=y +CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m CONFIG_VIRTUALIZATION=y CONFIG_KVM_BOOK3S_64=m CONFIG_KVM_BOOK3S_64_HV=m diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig index 0a8d250cb97e..58a98d40086f 100644 --- a/arch/powerpc/configs/ppc64_defconfig +++ b/arch/powerpc/configs/ppc64_defconfig @@ -10,6 +10,8 @@ CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y +CONFIG_LOG_BUF_SHIFT=18 +CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 CONFIG_CGROUPS=y CONFIG_CPUSETS=y CONFIG_BLK_DEV_INITRD=y @@ -90,7 +92,7 @@ CONFIG_BLK_DEV_AMD74XX=y CONFIG_BLK_DEV_IDE_PMAC=y CONFIG_BLK_DEV_IDE_PMAC_ATA100FIRST=y CONFIG_BLK_DEV_SD=y -CONFIG_CHR_DEV_ST=y +CONFIG_CHR_DEV_ST=m CONFIG_BLK_DEV_SR=y CONFIG_BLK_DEV_SR_VENDOR=y CONFIG_CHR_DEV_SG=y @@ -103,7 +105,7 @@ CONFIG_BE2ISCSI=m CONFIG_SCSI_MPT2SAS=m CONFIG_SCSI_IBMVSCSI=y CONFIG_SCSI_IBMVFC=m -CONFIG_SCSI_SYM53C8XX_2=y +CONFIG_SCSI_SYM53C8XX_2=m CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0 CONFIG_SCSI_IPR=y CONFIG_SCSI_QLA_FC=m @@ -149,10 +151,10 @@ CONFIG_NETCONSOLE=y CONFIG_TUN=m CONFIG_VIRTIO_NET=m CONFIG_VHOST_NET=m -CONFIG_VORTEX=y +CONFIG_VORTEX=m CONFIG_ACENIC=m CONFIG_ACENIC_OMIT_TIGON_I=y -CONFIG_PCNET32=y +CONFIG_PCNET32=m CONFIG_TIGON3=y CONFIG_BNX2X=m CONFIG_CHELSIO_T1=m @@ -165,6 +167,7 @@ CONFIG_E1000=y CONFIG_E1000E=y CONFIG_IXGB=m CONFIG_IXGBE=m +CONFIG_I40E=m CONFIG_MLX4_EN=m CONFIG_MYRI10GE=m CONFIG_PASEMI_MAC=y @@ -269,7 +272,7 @@ CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT4_FS=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y -CONFIG_REISERFS_FS=y +CONFIG_REISERFS_FS=m CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y @@ -284,10 +287,10 @@ CONFIG_NILFS2_FS=m CONFIG_AUTOFS4_FS=m CONFIG_FUSE_FS=m CONFIG_OVERLAY_FS=m -CONFIG_ISO9660_FS=y +CONFIG_ISO9660_FS=m CONFIG_UDF_FS=m CONFIG_MSDOS_FS=y -CONFIG_VFAT_FS=y +CONFIG_VFAT_FS=m CONFIG_PROC_KCORE=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y @@ -347,6 +350,8 @@ CONFIG_CRYPTO_LZO=m # CONFIG_CRYPTO_ANSI_CPRNG is not set CONFIG_CRYPTO_DEV_NX=y CONFIG_CRYPTO_DEV_NX_ENCRYPT=m +CONFIG_CRYPTO_DEV_VMX=y +CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m CONFIG_VIRTUALIZATION=y CONFIG_KVM_BOOK3S_64=m CONFIG_KVM_BOOK3S_64_HV=m diff --git a/arch/powerpc/configs/pseries_defconfig b/arch/powerpc/configs/pseries_defconfig index 654aeffc57ef..8a3bc016b732 100644 --- a/arch/powerpc/configs/pseries_defconfig +++ b/arch/powerpc/configs/pseries_defconfig @@ -15,6 +15,8 @@ CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y +CONFIG_LOG_BUF_SHIFT=18 +CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 CONFIG_NUMA_BALANCING=y CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y CONFIG_CGROUPS=y @@ -95,7 +97,7 @@ CONFIG_BLK_DEV_IDECD=y CONFIG_BLK_DEV_GENERIC=y CONFIG_BLK_DEV_AMD74XX=y CONFIG_BLK_DEV_SD=y -CONFIG_CHR_DEV_ST=y +CONFIG_CHR_DEV_ST=m CONFIG_BLK_DEV_SR=y CONFIG_BLK_DEV_SR_VENDOR=y CONFIG_CHR_DEV_SG=y @@ -108,7 +110,7 @@ CONFIG_BE2ISCSI=m CONFIG_SCSI_MPT2SAS=m CONFIG_SCSI_IBMVSCSI=y CONFIG_SCSI_IBMVFC=m -CONFIG_SCSI_SYM53C8XX_2=y +CONFIG_SCSI_SYM53C8XX_2=m CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0 CONFIG_SCSI_IPR=y CONFIG_SCSI_QLA_FC=m @@ -150,10 +152,10 @@ CONFIG_TUN=m CONFIG_VETH=m CONFIG_VIRTIO_NET=m CONFIG_VHOST_NET=m -CONFIG_VORTEX=y +CONFIG_VORTEX=m CONFIG_ACENIC=m CONFIG_ACENIC_OMIT_TIGON_I=y -CONFIG_PCNET32=y +CONFIG_PCNET32=m CONFIG_TIGON3=y CONFIG_BNX2X=m CONFIG_CHELSIO_T1=m @@ -166,6 +168,7 @@ CONFIG_E1000=y CONFIG_E1000E=y CONFIG_IXGB=m CONFIG_IXGBE=m +CONFIG_I40E=m CONFIG_MLX4_EN=m CONFIG_MYRI10GE=m CONFIG_QLGE=m @@ -241,7 +244,7 @@ CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT4_FS=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y -CONFIG_REISERFS_FS=y +CONFIG_REISERFS_FS=m CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y @@ -256,10 +259,10 @@ CONFIG_NILFS2_FS=m CONFIG_AUTOFS4_FS=m CONFIG_FUSE_FS=m CONFIG_OVERLAY_FS=m -CONFIG_ISO9660_FS=y +CONFIG_ISO9660_FS=m CONFIG_UDF_FS=m CONFIG_MSDOS_FS=y -CONFIG_VFAT_FS=y +CONFIG_VFAT_FS=m CONFIG_PROC_KCORE=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y @@ -314,6 +317,8 @@ CONFIG_CRYPTO_LZO=m # CONFIG_CRYPTO_ANSI_CPRNG is not set CONFIG_CRYPTO_DEV_NX=y CONFIG_CRYPTO_DEV_NX_ENCRYPT=m +CONFIG_CRYPTO_DEV_VMX=y +CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m CONFIG_VIRTUALIZATION=y CONFIG_KVM_BOOK3S_64=m CONFIG_KVM_BOOK3S_64_HV=m diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h index e71b9097594c..d1492736d852 100644 --- a/arch/powerpc/include/asm/asm-prototypes.h +++ b/arch/powerpc/include/asm/asm-prototypes.h @@ -15,6 +15,8 @@ #include <linux/threads.h> #include <linux/kprobes.h> +#include <uapi/asm/ucontext.h> + /* SMP */ extern struct thread_info *current_set[NR_CPUS]; extern struct thread_info *secondary_ti; @@ -52,8 +54,8 @@ void SMIException(struct pt_regs *regs); void handle_hmi_exception(struct pt_regs *regs); void instruction_breakpoint_exception(struct pt_regs *regs); void RunModeException(struct pt_regs *regs); -void __kprobes single_step_exception(struct pt_regs *regs); -void __kprobes program_check_exception(struct pt_regs *regs); +void single_step_exception(struct pt_regs *regs); +void program_check_exception(struct pt_regs *regs); void alignment_exception(struct pt_regs *regs); void StackOverflow(struct pt_regs *regs); void nonrecoverable_exception(struct pt_regs *regs); @@ -70,6 +72,41 @@ void unrecoverable_exception(struct pt_regs *regs); void kernel_bad_stack(struct pt_regs *regs); void system_reset_exception(struct pt_regs *regs); void machine_check_exception(struct pt_regs *regs); -void __kprobes emulation_assist_interrupt(struct pt_regs *regs); +void emulation_assist_interrupt(struct pt_regs *regs); + +/* signals, syscalls and interrupts */ +#ifdef CONFIG_PPC64 +int sys_swapcontext(struct ucontext __user *old_ctx, + struct ucontext __user *new_ctx, + long ctx_size, long r6, long r7, long r8, struct pt_regs *regs); +#else +long sys_swapcontext(struct ucontext __user *old_ctx, + struct ucontext __user *new_ctx, + int ctx_size, int r6, int r7, int r8, struct pt_regs *regs); +#endif +long sys_switch_endian(void); +notrace unsigned int __check_irq_replay(void); +void notrace restore_interrupts(void); + +/* ptrace */ +long do_syscall_trace_enter(struct pt_regs *regs); +void do_syscall_trace_leave(struct pt_regs *regs); + +/* process */ +void restore_math(struct pt_regs *regs); +void restore_tm_state(struct pt_regs *regs); + +/* prom_init (OpenFirmware) */ +unsigned long __init prom_init(unsigned long r3, unsigned long r4, + unsigned long pp, + unsigned long r6, unsigned long r7, + unsigned long kbase); + +/* setup */ +void __init early_setup(unsigned long dt_ptr); +void early_setup_secondary(void); + +/* time */ +void accumulate_stolen_time(void); #endif /* _ASM_POWERPC_ASM_PROTOTYPES_H */ diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index f08d567e0ca4..2b90335194a7 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -233,7 +233,7 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u) PPC_ATOMIC_ENTRY_BARRIER "1: lwarx %0,0,%1 # __atomic_add_unless\n\ cmpw 0,%0,%3 \n\ - beq- 2f \n\ + beq 2f \n\ add %0,%2,%0 \n" PPC405_ERR77(0,%2) " stwcx. %0,0,%1 \n\ @@ -539,7 +539,7 @@ static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u) PPC_ATOMIC_ENTRY_BARRIER "1: ldarx %0,0,%1 # __atomic_add_unless\n\ cmpd 0,%0,%3 \n\ - beq- 2f \n\ + beq 2f \n\ add %0,%2,%0 \n" " stdcx. %0,0,%1 \n\ bne- 1b \n" diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h index 38b33dcfcc9d..6b8b2d57fdc8 100644 --- a/arch/powerpc/include/asm/book3s/32/pgtable.h +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h @@ -223,7 +223,8 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, } -static inline void __ptep_set_access_flags(pte_t *ptep, pte_t entry) +static inline void __ptep_set_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 263bf39ced40..9fd77f8794a0 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -6,6 +6,8 @@ */ #define _PAGE_BIT_SWAP_TYPE 0 +#define _PAGE_RO 0 + #define _PAGE_EXEC 0x00001 /* execute permission */ #define _PAGE_WRITE 0x00002 /* write access allowed */ #define _PAGE_READ 0x00004 /* read access allowed */ @@ -565,10 +567,11 @@ static inline bool check_pte_access(unsigned long access, unsigned long ptev) * Generic functions with hash/radix callbacks */ -static inline void __ptep_set_access_flags(pte_t *ptep, pte_t entry) +static inline void __ptep_set_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { if (radix_enabled()) - return radix__ptep_set_access_flags(ptep, entry); + return radix__ptep_set_access_flags(mm, ptep, entry); return hash__ptep_set_access_flags(ptep, entry); } diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index df294224e280..2a46dea8e1b1 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -11,6 +11,11 @@ #include <asm/book3s/64/radix-4k.h> #endif +#ifndef __ASSEMBLY__ +#include <asm/book3s/64/tlbflush-radix.h> +#include <asm/cpu_has_feature.h> +#endif + /* An empty PTE can still have a R or C writeback */ #define RADIX_PTE_NONE_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) @@ -105,11 +110,8 @@ #define RADIX_PUD_TABLE_SIZE (sizeof(pud_t) << RADIX_PUD_INDEX_SIZE) #define RADIX_PGD_TABLE_SIZE (sizeof(pgd_t) << RADIX_PGD_INDEX_SIZE) -static inline unsigned long radix__pte_update(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, unsigned long clr, - unsigned long set, - int huge) +static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, + unsigned long set) { pte_t pte; unsigned long old_pte, new_pte; @@ -121,9 +123,39 @@ static inline unsigned long radix__pte_update(struct mm_struct *mm, } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); - /* We already do a sync in cmpxchg, is ptesync needed ?*/ + return old_pte; +} + + +static inline unsigned long radix__pte_update(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long clr, + unsigned long set, + int huge) +{ + unsigned long old_pte; + + if (cpu_has_feature(CPU_FTR_POWER9_DD1)) { + + unsigned long new_pte; + + old_pte = __radix_pte_update(ptep, ~0, 0); + asm volatile("ptesync" : : : "memory"); + /* + * new value of pte + */ + new_pte = (old_pte | set) & ~clr; + + /* + * For now let's do heavy pid flush + * radix__flush_tlb_page_psize(mm, addr, mmu_virtual_psize); + */ + radix__flush_tlb_mm(mm); + + __radix_pte_update(ptep, 0, new_pte); + } else + old_pte = __radix_pte_update(ptep, clr, set); asm volatile("ptesync" : : : "memory"); - /* huge pages use the old page table lock */ if (!huge) assert_pte_locked(mm, addr); @@ -134,20 +166,33 @@ static inline unsigned long radix__pte_update(struct mm_struct *mm, * Set the dirty and/or accessed bits atomically in a linux PTE, this * function doesn't need to invalidate tlb. */ -static inline void radix__ptep_set_access_flags(pte_t *ptep, pte_t entry) +static inline void radix__ptep_set_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { - pte_t pte; - unsigned long old_pte, new_pte; + unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); - do { - pte = READ_ONCE(*ptep); - old_pte = pte_val(pte); + + if (cpu_has_feature(CPU_FTR_POWER9_DD1)) { + + unsigned long old_pte, new_pte; + + old_pte = __radix_pte_update(ptep, ~0, 0); + asm volatile("ptesync" : : : "memory"); + /* + * new value of pte + */ new_pte = old_pte | set; - } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); + /* + * For now let's do heavy pid flush + * radix__flush_tlb_page_psize(mm, addr, mmu_virtual_psize); + */ + radix__flush_tlb_mm(mm); - /* We already do a sync in cmpxchg, is ptesync needed ?*/ + __radix_pte_update(ptep, 0, new_pte); + } else + __radix_pte_update(ptep, 0, set); asm volatile("ptesync" : : : "memory"); } @@ -233,14 +278,19 @@ static inline unsigned long radix__get_tree_size(void) { unsigned long rts_field; /* - * we support 52 bits, hence 52-31 = 21, 0b10101 + * We support 52 bits, hence: + * DD1 52-28 = 24, 0b11000 + * Others 52-31 = 21, 0b10101 * RTS encoding details * bits 0 - 3 of rts -> bits 6 - 8 unsigned long * bits 4 - 5 of rts -> bits 62 - 63 of unsigned long */ - rts_field = (0x5UL << 5); /* 6 - 8 bits */ - rts_field |= (0x2UL << 61); - + if (cpu_has_feature(CPU_FTR_POWER9_DD1)) + rts_field = (0x3UL << 61); + else { + rts_field = (0x5UL << 5); /* 6 - 8 bits */ + rts_field |= (0x2UL << 61); + } return rts_field; } #endif /* __ASSEMBLY__ */ diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 65037762b120..a9e19cb2f7c5 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -41,4 +41,5 @@ extern void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmad extern void radix__flush_tlb_lpid_va(unsigned long lpid, unsigned long gpa, unsigned long page_size); extern void radix__flush_tlb_lpid(unsigned long lpid); +extern void radix__flush_tlb_all(void); #endif diff --git a/arch/powerpc/include/asm/cputable.h b/arch/powerpc/include/asm/cputable.h index 82026b419341..f752e6f7cfbe 100644 --- a/arch/powerpc/include/asm/cputable.h +++ b/arch/powerpc/include/asm/cputable.h @@ -212,6 +212,7 @@ enum { #define CPU_FTR_DABRX LONG_ASM_CONST(0x0800000000000000) #define CPU_FTR_PMAO_BUG LONG_ASM_CONST(0x1000000000000000) #define CPU_FTR_SUBCORE LONG_ASM_CONST(0x2000000000000000) +#define CPU_FTR_POWER9_DD1 LONG_ASM_CONST(0x4000000000000000) #ifndef __ASSEMBLY__ @@ -472,6 +473,7 @@ enum { CPU_FTR_ICSWX | CPU_FTR_CFAR | CPU_FTR_HVMODE | CPU_FTR_VMX_COPY | \ CPU_FTR_DBELL | CPU_FTR_HAS_PPR | CPU_FTR_DAWR | \ CPU_FTR_ARCH_207S | CPU_FTR_TM_COMP | CPU_FTR_ARCH_300) +#define CPU_FTRS_POWER9_DD1 (CPU_FTRS_POWER9 | CPU_FTR_POWER9_DD1) #define CPU_FTRS_CELL (CPU_FTR_USE_TB | CPU_FTR_LWSYNC | \ CPU_FTR_PPCAS_ARCH_V2 | CPU_FTR_CTRL | \ CPU_FTR_ALTIVEC_COMP | CPU_FTR_MMCRA | CPU_FTR_SMT | \ @@ -490,7 +492,7 @@ enum { (CPU_FTRS_POWER4 | CPU_FTRS_PPC970 | CPU_FTRS_POWER5 | \ CPU_FTRS_POWER6 | CPU_FTRS_POWER7 | CPU_FTRS_POWER8E | \ CPU_FTRS_POWER8 | CPU_FTRS_POWER8_DD1 | CPU_FTRS_CELL | \ - CPU_FTRS_PA6T | CPU_FTR_VSX | CPU_FTRS_POWER9) + CPU_FTRS_PA6T | CPU_FTR_VSX | CPU_FTRS_POWER9 | CPU_FTRS_POWER9_DD1) #endif #else enum { diff --git a/arch/powerpc/include/asm/exception-64s.h b/arch/powerpc/include/asm/exception-64s.h index bed66e5743b3..2e4e7d878c8e 100644 --- a/arch/powerpc/include/asm/exception-64s.h +++ b/arch/powerpc/include/asm/exception-64s.h @@ -34,6 +34,7 @@ * exception handlers (including pSeries LPAR) and iSeries LPAR * implementations as possible. */ +#include <asm/head-64.h> #define EX_R9 0 #define EX_R10 8 @@ -52,7 +53,6 @@ #ifdef CONFIG_RELOCATABLE #define __EXCEPTION_RELON_PROLOG_PSERIES_1(label, h) \ - ld r12,PACAKBASE(r13); /* get high part of &label */ \ mfspr r11,SPRN_##h##SRR0; /* save SRR0 */ \ LOAD_HANDLER(r12,label); \ mtctr r12; \ @@ -84,13 +84,14 @@ /* * We're short on space and time in the exception prolog, so we can't - * use the normal SET_REG_IMMEDIATE macro. Normally we just need the - * low halfword of the address, but for Kdump we need the whole low - * word. + * use the normal LOAD_REG_IMMEDIATE macro to load the address of label. + * Instead we get the base of the kernel from paca->kernelbase and or in the low + * part of label. This requires that the label be within 64KB of kernelbase, and + * that kernelbase be 64K aligned. */ #define LOAD_HANDLER(reg, label) \ - /* Handlers must be within 64K of kbase, which must be 64k aligned */ \ - ori reg,reg,(label)-_stext; /* virt addr of handler ... */ + ld reg,PACAKBASE(r13); /* get high part of &label */ \ + ori reg,reg,(FIXED_SYMBOL_ABS_ADDR(label))@l; /* Exception register prefixes */ #define EXC_HV H @@ -175,7 +176,6 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) __EXCEPTION_PROLOG_1(area, extra, vec) #define __EXCEPTION_PROLOG_PSERIES_1(label, h) \ - ld r12,PACAKBASE(r13); /* get high part of &label */ \ ld r10,PACAKMSR(r13); /* get MSR value for kernel */ \ mfspr r11,SPRN_##h##SRR0; /* save SRR0 */ \ LOAD_HANDLER(r12,label) \ @@ -192,10 +192,10 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) EXCEPTION_PROLOG_1(area, extra, vec); \ EXCEPTION_PROLOG_PSERIES_1(label, h); -#define __KVMTEST(n) \ - lbz r10,HSTATE_IN_GUEST(r13); \ +#define __KVMTEST(h, n) \ + lbz r10,HSTATE_IN_GUEST(r13); \ cmpwi r10,0; \ - bne do_kvm_##n + bne do_kvm_##h##n #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE /* @@ -208,8 +208,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) #define kvmppc_interrupt kvmppc_interrupt_pr #endif -#define __KVM_HANDLER(area, h, n) \ -do_kvm_##n: \ +#define __KVM_HANDLER_PROLOG(area, n) \ BEGIN_FTR_SECTION_NESTED(947) \ ld r10,area+EX_CFAR(r13); \ std r10,HSTATE_CFAR(r13); \ @@ -222,21 +221,23 @@ do_kvm_##n: \ stw r9,HSTATE_SCRATCH1(r13); \ ld r9,area+EX_R9(r13); \ std r12,HSTATE_SCRATCH0(r13); \ + +#define __KVM_HANDLER(area, h, n) \ + __KVM_HANDLER_PROLOG(area, n) \ li r12,n; \ b kvmppc_interrupt #define __KVM_HANDLER_SKIP(area, h, n) \ -do_kvm_##n: \ cmpwi r10,KVM_GUEST_MODE_SKIP; \ ld r10,area+EX_R10(r13); \ beq 89f; \ - stw r9,HSTATE_SCRATCH1(r13); \ + stw r9,HSTATE_SCRATCH1(r13); \ BEGIN_FTR_SECTION_NESTED(948) \ ld r9,area+EX_PPR(r13); \ std r9,HSTATE_PPR(r13); \ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948); \ ld r9,area+EX_R9(r13); \ - std r12,HSTATE_SCRATCH0(r13); \ + std r12,HSTATE_SCRATCH0(r13); \ li r12,n; \ b kvmppc_interrupt; \ 89: mtocrf 0x80,r9; \ @@ -244,12 +245,12 @@ do_kvm_##n: \ b kvmppc_skip_##h##interrupt #ifdef CONFIG_KVM_BOOK3S_64_HANDLER -#define KVMTEST(n) __KVMTEST(n) +#define KVMTEST(h, n) __KVMTEST(h, n) #define KVM_HANDLER(area, h, n) __KVM_HANDLER(area, h, n) #define KVM_HANDLER_SKIP(area, h, n) __KVM_HANDLER_SKIP(area, h, n) #else -#define KVMTEST(n) +#define KVMTEST(h, n) #define KVM_HANDLER(area, h, n) #define KVM_HANDLER_SKIP(area, h, n) #endif @@ -333,94 +334,79 @@ do_kvm_##n: \ /* * Exception vectors. */ -#define STD_EXCEPTION_PSERIES(vec, label) \ - . = vec; \ - .globl label##_pSeries; \ -label##_pSeries: \ +#define STD_EXCEPTION_PSERIES(vec, label) \ SET_SCRATCH0(r13); /* save r13 */ \ - EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, label##_common, \ - EXC_STD, KVMTEST, vec) + EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, label, \ + EXC_STD, KVMTEST_PR, vec); \ /* Version of above for when we have to branch out-of-line */ +#define __OOL_EXCEPTION(vec, label, hdlr) \ + SET_SCRATCH0(r13) \ + EXCEPTION_PROLOG_0(PACA_EXGEN) \ + b hdlr; + #define STD_EXCEPTION_PSERIES_OOL(vec, label) \ - .globl label##_pSeries; \ -label##_pSeries: \ - EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST, vec); \ - EXCEPTION_PROLOG_PSERIES_1(label##_common, EXC_STD) - -#define STD_EXCEPTION_HV(loc, vec, label) \ - . = loc; \ - .globl label##_hv; \ -label##_hv: \ + EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_PR, vec); \ + EXCEPTION_PROLOG_PSERIES_1(label, EXC_STD) + +#define STD_EXCEPTION_HV(loc, vec, label) \ SET_SCRATCH0(r13); /* save r13 */ \ - EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, label##_common, \ - EXC_HV, KVMTEST, vec) + EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, label, \ + EXC_HV, KVMTEST_HV, vec); -/* Version of above for when we have to branch out-of-line */ -#define STD_EXCEPTION_HV_OOL(vec, label) \ - .globl label##_hv; \ -label##_hv: \ - EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST, vec); \ - EXCEPTION_PROLOG_PSERIES_1(label##_common, EXC_HV) +#define STD_EXCEPTION_HV_OOL(vec, label) \ + EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_HV, vec); \ + EXCEPTION_PROLOG_PSERIES_1(label, EXC_HV) #define STD_RELON_EXCEPTION_PSERIES(loc, vec, label) \ - . = loc; \ - .globl label##_relon_pSeries; \ -label##_relon_pSeries: \ /* No guest interrupts come through here */ \ SET_SCRATCH0(r13); /* save r13 */ \ - EXCEPTION_RELON_PROLOG_PSERIES(PACA_EXGEN, label##_common, \ - EXC_STD, NOTEST, vec) + EXCEPTION_RELON_PROLOG_PSERIES(PACA_EXGEN, label, EXC_STD, NOTEST, vec); #define STD_RELON_EXCEPTION_PSERIES_OOL(vec, label) \ - .globl label##_relon_pSeries; \ -label##_relon_pSeries: \ EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, vec); \ - EXCEPTION_RELON_PROLOG_PSERIES_1(label##_common, EXC_STD) + EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_STD) #define STD_RELON_EXCEPTION_HV(loc, vec, label) \ - . = loc; \ - .globl label##_relon_hv; \ -label##_relon_hv: \ /* No guest interrupts come through here */ \ SET_SCRATCH0(r13); /* save r13 */ \ - EXCEPTION_RELON_PROLOG_PSERIES(PACA_EXGEN, label##_common, \ - EXC_HV, NOTEST, vec) + EXCEPTION_RELON_PROLOG_PSERIES(PACA_EXGEN, label, EXC_HV, NOTEST, vec); #define STD_RELON_EXCEPTION_HV_OOL(vec, label) \ - .globl label##_relon_hv; \ -label##_relon_hv: \ EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, vec); \ - EXCEPTION_RELON_PROLOG_PSERIES_1(label##_common, EXC_HV) + EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) /* This associate vector numbers with bits in paca->irq_happened */ #define SOFTEN_VALUE_0x500 PACA_IRQ_EE -#define SOFTEN_VALUE_0x502 PACA_IRQ_EE #define SOFTEN_VALUE_0x900 PACA_IRQ_DEC -#define SOFTEN_VALUE_0x982 PACA_IRQ_DEC +#define SOFTEN_VALUE_0x980 PACA_IRQ_DEC #define SOFTEN_VALUE_0xa00 PACA_IRQ_DBELL #define SOFTEN_VALUE_0xe80 PACA_IRQ_DBELL -#define SOFTEN_VALUE_0xe82 PACA_IRQ_DBELL #define SOFTEN_VALUE_0xe60 PACA_IRQ_HMI -#define SOFTEN_VALUE_0xe62 PACA_IRQ_HMI #define SOFTEN_VALUE_0xea0 PACA_IRQ_EE -#define SOFTEN_VALUE_0xea2 PACA_IRQ_EE #define __SOFTEN_TEST(h, vec) \ lbz r10,PACASOFTIRQEN(r13); \ cmpwi r10,0; \ li r10,SOFTEN_VALUE_##vec; \ beq masked_##h##interrupt + #define _SOFTEN_TEST(h, vec) __SOFTEN_TEST(h, vec) #define SOFTEN_TEST_PR(vec) \ - KVMTEST(vec); \ + KVMTEST(EXC_STD, vec); \ _SOFTEN_TEST(EXC_STD, vec) #define SOFTEN_TEST_HV(vec) \ - KVMTEST(vec); \ + KVMTEST(EXC_HV, vec); \ _SOFTEN_TEST(EXC_HV, vec) +#define KVMTEST_PR(vec) \ + KVMTEST(EXC_STD, vec) + +#define KVMTEST_HV(vec) \ + KVMTEST(EXC_HV, vec) + #define SOFTEN_NOTEST_PR(vec) _SOFTEN_TEST(EXC_STD, vec) #define SOFTEN_NOTEST_HV(vec) _SOFTEN_TEST(EXC_HV, vec) @@ -428,58 +414,47 @@ label##_relon_hv: \ SET_SCRATCH0(r13); /* save r13 */ \ EXCEPTION_PROLOG_0(PACA_EXGEN); \ __EXCEPTION_PROLOG_1(PACA_EXGEN, extra, vec); \ - EXCEPTION_PROLOG_PSERIES_1(label##_common, h); + EXCEPTION_PROLOG_PSERIES_1(label, h); #define _MASKABLE_EXCEPTION_PSERIES(vec, label, h, extra) \ __MASKABLE_EXCEPTION_PSERIES(vec, label, h, extra) #define MASKABLE_EXCEPTION_PSERIES(loc, vec, label) \ - . = loc; \ - .globl label##_pSeries; \ -label##_pSeries: \ _MASKABLE_EXCEPTION_PSERIES(vec, label, \ EXC_STD, SOFTEN_TEST_PR) +#define MASKABLE_EXCEPTION_PSERIES_OOL(vec, label) \ + EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_TEST_PR, vec); \ + EXCEPTION_PROLOG_PSERIES_1(label, EXC_STD) + #define MASKABLE_EXCEPTION_HV(loc, vec, label) \ - . = loc; \ - .globl label##_hv; \ -label##_hv: \ _MASKABLE_EXCEPTION_PSERIES(vec, label, \ EXC_HV, SOFTEN_TEST_HV) #define MASKABLE_EXCEPTION_HV_OOL(vec, label) \ - .globl label##_hv; \ -label##_hv: \ EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_TEST_HV, vec); \ - EXCEPTION_PROLOG_PSERIES_1(label##_common, EXC_HV); + EXCEPTION_PROLOG_PSERIES_1(label, EXC_HV) #define __MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, h, extra) \ SET_SCRATCH0(r13); /* save r13 */ \ EXCEPTION_PROLOG_0(PACA_EXGEN); \ - __EXCEPTION_PROLOG_1(PACA_EXGEN, extra, vec); \ - EXCEPTION_RELON_PROLOG_PSERIES_1(label##_common, h); -#define _MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, h, extra) \ + __EXCEPTION_PROLOG_1(PACA_EXGEN, extra, vec); \ + EXCEPTION_RELON_PROLOG_PSERIES_1(label, h) + +#define _MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, h, extra) \ __MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, h, extra) #define MASKABLE_RELON_EXCEPTION_PSERIES(loc, vec, label) \ - . = loc; \ - .globl label##_relon_pSeries; \ -label##_relon_pSeries: \ _MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, \ EXC_STD, SOFTEN_NOTEST_PR) #define MASKABLE_RELON_EXCEPTION_HV(loc, vec, label) \ - . = loc; \ - .globl label##_relon_hv; \ -label##_relon_hv: \ _MASKABLE_RELON_EXCEPTION_PSERIES(vec, label, \ EXC_HV, SOFTEN_NOTEST_HV) #define MASKABLE_RELON_EXCEPTION_HV_OOL(vec, label) \ - .globl label##_relon_hv; \ -label##_relon_hv: \ EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_NOTEST_HV, vec); \ - EXCEPTION_PROLOG_PSERIES_1(label##_common, EXC_HV); + EXCEPTION_PROLOG_PSERIES_1(label, EXC_HV) /* * Our exception common code can be passed various "additions" @@ -505,9 +480,6 @@ BEGIN_FTR_SECTION \ END_FTR_SECTION_IFSET(CPU_FTR_CTRL) #define EXCEPTION_COMMON(trap, label, hdlr, ret, additions) \ - .align 7; \ - .globl label##_common; \ -label##_common: \ EXCEPTION_PROLOG_COMMON(trap, PACA_EXGEN); \ /* Volatile regs are potentially clobbered here */ \ additions; \ diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h index b4407d0add27..0031806475f0 100644 --- a/arch/powerpc/include/asm/fadump.h +++ b/arch/powerpc/include/asm/fadump.h @@ -45,10 +45,6 @@ #define memblock_num_regions(memblock_type) (memblock.memblock_type.cnt) -#ifndef ELF_CORE_EFLAGS -#define ELF_CORE_EFLAGS 0 -#endif - /* Firmware provided dump sections */ #define FADUMP_CPU_STATE_DATA 0x0001 #define FADUMP_HPTE_REGION 0x0002 diff --git a/arch/powerpc/include/asm/head-64.h b/arch/powerpc/include/asm/head-64.h new file mode 100644 index 000000000000..ab90c2fa1ea6 --- /dev/null +++ b/arch/powerpc/include/asm/head-64.h @@ -0,0 +1,393 @@ +#ifndef _ASM_POWERPC_HEAD_64_H +#define _ASM_POWERPC_HEAD_64_H + +#include <asm/cache.h> + +/* + * We can't do CPP stringification and concatination directly into the section + * name for some reason, so these macros can do it for us. + */ +.macro define_ftsec name + .section ".head.text.\name\()","ax",@progbits +.endm +.macro define_data_ftsec name + .section ".head.data.\name\()","a",@progbits +.endm +.macro use_ftsec name + .section ".head.text.\name\()" +.endm + +/* + * Fixed (location) sections are used by opening fixed sections and emitting + * fixed section entries into them before closing them. Multiple fixed sections + * can be open at any time. + * + * Each fixed section created in a .S file must have corresponding linkage + * directives including location, added to arch/powerpc/kernel/vmlinux.lds.S + * + * For each fixed section, code is generated into it in the order which it + * appears in the source. Fixed section entries can be placed at a fixed + * location within the section using _LOCATION postifx variants. These must + * be ordered according to their relative placements within the section. + * + * OPEN_FIXED_SECTION(section_name, start_address, end_address) + * FIXED_SECTION_ENTRY_BEGIN(section_name, label1) + * + * USE_FIXED_SECTION(section_name) + * label3: + * li r10,128 + * mv r11,r10 + + * FIXED_SECTION_ENTRY_BEGIN_LOCATION(section_name, label2, start_address) + * FIXED_SECTION_ENTRY_END_LOCATION(section_name, label2, end_address) + * CLOSE_FIXED_SECTION(section_name) + * + * ZERO_FIXED_SECTION can be used to emit zeroed data. + * + * Troubleshooting: + * - If the build dies with "Error: attempt to move .org backwards" at + * CLOSE_FIXED_SECTION() or elsewhere, there may be something + * unexpected being added there. Remove the '. = x_len' line, rebuild, and + * check what is pushing the section down. + * - If the build dies in linking, check arch/powerpc/kernel/vmlinux.lds.S + * for instructions. + * - If the kernel crashes or hangs in very early boot, it could be linker + * stubs at the start of the main text. + */ + +#define OPEN_FIXED_SECTION(sname, start, end) \ + sname##_start = (start); \ + sname##_end = (end); \ + sname##_len = (end) - (start); \ + define_ftsec sname; \ + . = 0x0; \ +start_##sname: + +#define OPEN_TEXT_SECTION(start) \ + text_start = (start); \ + .section ".text","ax",@progbits; \ + . = 0x0; \ +start_text: + +#define ZERO_FIXED_SECTION(sname, start, end) \ + sname##_start = (start); \ + sname##_end = (end); \ + sname##_len = (end) - (start); \ + define_data_ftsec sname; \ + . = 0x0; \ + . = sname##_len; + +#define USE_FIXED_SECTION(sname) \ + fs_label = start_##sname; \ + fs_start = sname##_start; \ + use_ftsec sname; + +#define USE_TEXT_SECTION() \ + fs_label = start_text; \ + fs_start = text_start; \ + .text + +#define CLOSE_FIXED_SECTION(sname) \ + USE_FIXED_SECTION(sname); \ + . = sname##_len; \ +end_##sname: + + +#define __FIXED_SECTION_ENTRY_BEGIN(sname, name, __align) \ + USE_FIXED_SECTION(sname); \ + .align __align; \ + .global name; \ +name: + +#define FIXED_SECTION_ENTRY_BEGIN(sname, name) \ + __FIXED_SECTION_ENTRY_BEGIN(sname, name, 0) + +#define FIXED_SECTION_ENTRY_BEGIN_LOCATION(sname, name, start) \ + USE_FIXED_SECTION(sname); \ + name##_start = (start); \ + .if (start) < sname##_start; \ + .error "Fixed section underflow"; \ + .abort; \ + .endif; \ + . = (start) - sname##_start; \ + .global name; \ +name: + +#define FIXED_SECTION_ENTRY_END_LOCATION(sname, name, end) \ + .if (end) > sname##_end; \ + .error "Fixed section overflow"; \ + .abort; \ + .endif; \ + .if (. - name > end - name##_start); \ + .error "Fixed entry overflow"; \ + .abort; \ + .endif; \ + . = ((end) - sname##_start); \ + + +/* + * These macros are used to change symbols in other fixed sections to be + * absolute or related to our current fixed section. + * + * - DEFINE_FIXED_SYMBOL / FIXED_SYMBOL_ABS_ADDR is used to find the + * absolute address of a symbol within a fixed section, from any section. + * + * - ABS_ADDR is used to find the absolute address of any symbol, from within + * a fixed section. + */ +#define DEFINE_FIXED_SYMBOL(label) \ + label##_absolute = (label - fs_label + fs_start) + +#define FIXED_SYMBOL_ABS_ADDR(label) \ + (label##_absolute) + +#define ABS_ADDR(label) (label - fs_label + fs_start) + +/* + * Following are the BOOK3S exception handler helper macros. + * Handlers come in a number of types, and each type has a number of varieties. + * + * EXC_REAL_* - real, unrelocated exception vectors + * EXC_VIRT_* - virt (AIL), unrelocated exception vectors + * TRAMP_REAL_* - real, unrelocated helpers (virt can call these) + * TRAMP_VIRT_* - virt, unreloc helpers (in practice, real can use) + * TRAMP_KVM - KVM handlers that get put into real, unrelocated + * EXC_COMMON_* - virt, relocated common handlers + * + * The EXC handlers are given a name, and branch to name_common, or the + * appropriate KVM or masking function. Vector handler verieties are as + * follows: + * + * EXC_{REAL|VIRT}_BEGIN/END - used to open-code the exception + * + * EXC_{REAL|VIRT} - standard exception + * + * EXC_{REAL|VIRT}_suffix + * where _suffix is: + * - _MASKABLE - maskable exception + * - _OOL - out of line with trampoline to common handler + * - _HV - HV exception + * + * There can be combinations, e.g., EXC_VIRT_OOL_MASKABLE_HV + * + * The one unusual case is __EXC_REAL_OOL_HV_DIRECT, which is + * an OOL vector that branches to a specified handler rather than the usual + * trampoline that goes to common. It, and other underscore macros, should + * be used with care. + * + * KVM handlers come in the following verieties: + * TRAMP_KVM + * TRAMP_KVM_SKIP + * TRAMP_KVM_HV + * TRAMP_KVM_HV_SKIP + * + * COMMON handlers come in the following verieties: + * EXC_COMMON_BEGIN/END - used to open-code the handler + * EXC_COMMON + * EXC_COMMON_ASYNC + * EXC_COMMON_HV + * + * TRAMP_REAL and TRAMP_VIRT can be used with BEGIN/END. KVM + * and OOL handlers are implemented as types of TRAMP and TRAMP_VIRT handlers. + */ + +#define EXC_REAL_BEGIN(name, start, end) \ + FIXED_SECTION_ENTRY_BEGIN_LOCATION(real_vectors, exc_real_##start##_##name, start) + +#define EXC_REAL_END(name, start, end) \ + FIXED_SECTION_ENTRY_END_LOCATION(real_vectors, exc_real_##start##_##name, end) + +#define EXC_VIRT_BEGIN(name, start, end) \ + FIXED_SECTION_ENTRY_BEGIN_LOCATION(virt_vectors, exc_virt_##start##_##name, start) + +#define EXC_VIRT_END(name, start, end) \ + FIXED_SECTION_ENTRY_END_LOCATION(virt_vectors, exc_virt_##start##_##name, end) + +#define EXC_COMMON_BEGIN(name) \ + USE_TEXT_SECTION(); \ + .align 7; \ + .global name; \ + DEFINE_FIXED_SYMBOL(name); \ +name: + +#define TRAMP_REAL_BEGIN(name) \ + FIXED_SECTION_ENTRY_BEGIN(real_trampolines, name) + +#define TRAMP_VIRT_BEGIN(name) \ + FIXED_SECTION_ENTRY_BEGIN(virt_trampolines, name) + +#ifdef CONFIG_KVM_BOOK3S_64_HANDLER +#define TRAMP_KVM_BEGIN(name) \ + TRAMP_REAL_BEGIN(name) +#else +#define TRAMP_KVM_BEGIN(name) +#endif + +#define EXC_REAL_NONE(start, end) \ + FIXED_SECTION_ENTRY_BEGIN_LOCATION(real_vectors, exc_real_##start##_##unused, start); \ + FIXED_SECTION_ENTRY_END_LOCATION(real_vectors, exc_real_##start##_##unused, end) + +#define EXC_VIRT_NONE(start, end) \ + FIXED_SECTION_ENTRY_BEGIN_LOCATION(virt_vectors, exc_virt_##start##_##unused, start); \ + FIXED_SECTION_ENTRY_END_LOCATION(virt_vectors, exc_virt_##start##_##unused, end); + + +#define EXC_REAL(name, start, end) \ + EXC_REAL_BEGIN(name, start, end); \ + STD_EXCEPTION_PSERIES(start, name##_common); \ + EXC_REAL_END(name, start, end); + +#define EXC_VIRT(name, start, end, realvec) \ + EXC_VIRT_BEGIN(name, start, end); \ + STD_RELON_EXCEPTION_PSERIES(start, realvec, name##_common); \ + EXC_VIRT_END(name, start, end); + +#define EXC_REAL_MASKABLE(name, start, end) \ + EXC_REAL_BEGIN(name, start, end); \ + MASKABLE_EXCEPTION_PSERIES(start, start, name##_common); \ + EXC_REAL_END(name, start, end); + +#define EXC_VIRT_MASKABLE(name, start, end, realvec) \ + EXC_VIRT_BEGIN(name, start, end); \ + MASKABLE_RELON_EXCEPTION_PSERIES(start, realvec, name##_common); \ + EXC_VIRT_END(name, start, end); + +#define EXC_REAL_HV(name, start, end) \ + EXC_REAL_BEGIN(name, start, end); \ + STD_EXCEPTION_HV(start, start, name##_common); \ + EXC_REAL_END(name, start, end); + +#define EXC_VIRT_HV(name, start, end, realvec) \ + EXC_VIRT_BEGIN(name, start, end); \ + STD_RELON_EXCEPTION_HV(start, realvec, name##_common); \ + EXC_VIRT_END(name, start, end); + +#define __EXC_REAL_OOL(name, start, end) \ + EXC_REAL_BEGIN(name, start, end); \ + __OOL_EXCEPTION(start, label, tramp_real_##name); \ + EXC_REAL_END(name, start, end); + +#define __TRAMP_REAL_REAL_OOL(name, vec) \ + TRAMP_REAL_BEGIN(tramp_real_##name); \ + STD_EXCEPTION_PSERIES_OOL(vec, name##_common); \ + +#define EXC_REAL_OOL(name, start, end) \ + __EXC_REAL_OOL(name, start, end); \ + __TRAMP_REAL_REAL_OOL(name, start); + +#define __EXC_REAL_OOL_MASKABLE(name, start, end) \ + __EXC_REAL_OOL(name, start, end); + +#define __TRAMP_REAL_REAL_OOL_MASKABLE(name, vec) \ + TRAMP_REAL_BEGIN(tramp_real_##name); \ + MASKABLE_EXCEPTION_PSERIES_OOL(vec, name##_common); \ + +#define EXC_REAL_OOL_MASKABLE(name, start, end) \ + __EXC_REAL_OOL_MASKABLE(name, start, end); \ + __TRAMP_REAL_REAL_OOL_MASKABLE(name, start); + +#define __EXC_REAL_OOL_HV_DIRECT(name, start, end, handler) \ + EXC_REAL_BEGIN(name, start, end); \ + __OOL_EXCEPTION(start, label, handler); \ + EXC_REAL_END(name, start, end); + +#define __EXC_REAL_OOL_HV(name, start, end) \ + __EXC_REAL_OOL(name, start, end); + +#define __TRAMP_REAL_REAL_OOL_HV(name, vec) \ + TRAMP_REAL_BEGIN(tramp_real_##name); \ + STD_EXCEPTION_HV_OOL(vec, name##_common); \ + +#define EXC_REAL_OOL_HV(name, start, end) \ + __EXC_REAL_OOL_HV(name, start, end); \ + __TRAMP_REAL_REAL_OOL_HV(name, start); + +#define __EXC_REAL_OOL_MASKABLE_HV(name, start, end) \ + __EXC_REAL_OOL(name, start, end); + +#define __TRAMP_REAL_REAL_OOL_MASKABLE_HV(name, vec) \ + TRAMP_REAL_BEGIN(tramp_real_##name); \ + MASKABLE_EXCEPTION_HV_OOL(vec, name##_common); \ + +#define EXC_REAL_OOL_MASKABLE_HV(name, start, end) \ + __EXC_REAL_OOL_MASKABLE_HV(name, start, end); \ + __TRAMP_REAL_REAL_OOL_MASKABLE_HV(name, start); + +#define __EXC_VIRT_OOL(name, start, end) \ + EXC_VIRT_BEGIN(name, start, end); \ + __OOL_EXCEPTION(start, label, tramp_virt_##name); \ + EXC_VIRT_END(name, start, end); + +#define __TRAMP_REAL_VIRT_OOL(name, realvec) \ + TRAMP_VIRT_BEGIN(tramp_virt_##name); \ + STD_RELON_EXCEPTION_PSERIES_OOL(realvec, name##_common); \ + +#define EXC_VIRT_OOL(name, start, end, realvec) \ + __EXC_VIRT_OOL(name, start, end); \ + __TRAMP_REAL_VIRT_OOL(name, realvec); + +#define __EXC_VIRT_OOL_MASKABLE(name, start, end) \ + __EXC_VIRT_OOL(name, start, end); + +#define __TRAMP_REAL_VIRT_OOL_MASKABLE(name, realvec) \ + TRAMP_VIRT_BEGIN(tramp_virt_##name); \ + MASKABLE_RELON_EXCEPTION_PSERIES_OOL(realvec, name##_common); \ + +#define EXC_VIRT_OOL_MASKABLE(name, start, end, realvec) \ + __EXC_VIRT_OOL_MASKABLE(name, start, end); \ + __TRAMP_REAL_VIRT_OOL_MASKABLE(name, realvec); + +#define __EXC_VIRT_OOL_HV(name, start, end) \ + __EXC_VIRT_OOL(name, start, end); + +#define __TRAMP_REAL_VIRT_OOL_HV(name, realvec) \ + TRAMP_VIRT_BEGIN(tramp_virt_##name); \ + STD_RELON_EXCEPTION_HV_OOL(realvec, name##_common); \ + +#define EXC_VIRT_OOL_HV(name, start, end, realvec) \ + __EXC_VIRT_OOL_HV(name, start, end); \ + __TRAMP_REAL_VIRT_OOL_HV(name, realvec); + +#define __EXC_VIRT_OOL_MASKABLE_HV(name, start, end) \ + __EXC_VIRT_OOL(name, start, end); + +#define __TRAMP_REAL_VIRT_OOL_MASKABLE_HV(name, realvec) \ + TRAMP_VIRT_BEGIN(tramp_virt_##name); \ + MASKABLE_RELON_EXCEPTION_HV_OOL(realvec, name##_common); \ + +#define EXC_VIRT_OOL_MASKABLE_HV(name, start, end, realvec) \ + __EXC_VIRT_OOL_MASKABLE_HV(name, start, end); \ + __TRAMP_REAL_VIRT_OOL_MASKABLE_HV(name, realvec); + +#define TRAMP_KVM(area, n) \ + TRAMP_KVM_BEGIN(do_kvm_##n); \ + KVM_HANDLER(area, EXC_STD, n); \ + +#define TRAMP_KVM_SKIP(area, n) \ + TRAMP_KVM_BEGIN(do_kvm_##n); \ + KVM_HANDLER_SKIP(area, EXC_STD, n); \ + +/* + * HV variant exceptions get the 0x2 bit added to their trap number. + */ +#define TRAMP_KVM_HV(area, n) \ + TRAMP_KVM_BEGIN(do_kvm_H##n); \ + KVM_HANDLER(area, EXC_HV, n + 0x2); \ + +#define TRAMP_KVM_HV_SKIP(area, n) \ + TRAMP_KVM_BEGIN(do_kvm_H##n); \ + KVM_HANDLER_SKIP(area, EXC_HV, n + 0x2); \ + +#define EXC_COMMON(name, realvec, hdlr) \ + EXC_COMMON_BEGIN(name); \ + STD_EXCEPTION_COMMON(realvec, name, hdlr); \ + +#define EXC_COMMON_ASYNC(name, realvec, hdlr) \ + EXC_COMMON_BEGIN(name); \ + STD_EXCEPTION_COMMON_ASYNC(realvec, name, hdlr); \ + +#define EXC_COMMON_HV(name, realvec, hdlr) \ + EXC_COMMON_BEGIN(name); \ + STD_EXCEPTION_COMMON(realvec + 0x2, name, hdlr); \ + +#endif /* _ASM_POWERPC_HEAD_64_H */ diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h index 0420b388dd83..e02cbc6a6c70 100644 --- a/arch/powerpc/include/asm/machdep.h +++ b/arch/powerpc/include/asm/machdep.h @@ -61,7 +61,7 @@ struct machdep_calls { void (*init_IRQ)(void); - /* Return an irq, or NO_IRQ to indicate there are none pending. */ + /* Return an irq, or 0 to indicate there are none pending. */ unsigned int (*get_irq)(void); /* PCI stuff */ diff --git a/arch/powerpc/include/asm/mmu-book3e.h b/arch/powerpc/include/asm/mmu-book3e.h index cd4f04a74802..b62a8d43a06c 100644 --- a/arch/powerpc/include/asm/mmu-book3e.h +++ b/arch/powerpc/include/asm/mmu-book3e.h @@ -313,6 +313,9 @@ extern int book3e_htw_mode; * return 1, indicating that the tlb requires preloading. */ #define HUGETLB_NEED_PRELOAD + +#define mmu_cleanup_all NULL + #endif #endif /* !__ASSEMBLY__ */ diff --git a/arch/powerpc/include/asm/mmu.h b/arch/powerpc/include/asm/mmu.h index b78e8d3377f6..e88368354e49 100644 --- a/arch/powerpc/include/asm/mmu.h +++ b/arch/powerpc/include/asm/mmu.h @@ -204,6 +204,10 @@ extern unsigned int __start___mmu_ftr_fixup, __stop___mmu_ftr_fixup; * make it match the size our of bolted TLB area */ extern u64 ppc64_rma_size; + +/* Cleanup function used by kexec */ +extern void mmu_cleanup_all(void); +extern void radix__mmu_cleanup_all(void); #endif /* CONFIG_PPC64 */ struct mm_struct; diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 9d2cd0c36ec2..5c451140660a 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -18,6 +18,7 @@ extern void destroy_context(struct mm_struct *mm); #ifdef CONFIG_SPAPR_TCE_IOMMU struct mm_iommu_table_group_mem_t; +extern int isolate_lru_page(struct page *page); /* from internal.h */ extern bool mm_iommu_preregistered(void); extern long mm_iommu_get(unsigned long ua, unsigned long entries, struct mm_iommu_table_group_mem_t **pmem); @@ -71,7 +72,8 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk) { /* Mark this context has been used on the new CPU */ - cpumask_set_cpu(smp_processor_id(), mm_cpumask(next)); + if (!cpumask_test_cpu(smp_processor_id(), mm_cpumask(next))) + cpumask_set_cpu(smp_processor_id(), mm_cpumask(next)); /* 32-bit keeps track of the current PGDIR in the thread struct */ #ifdef CONFIG_PPC32 diff --git a/arch/powerpc/include/asm/mpic_msgr.h b/arch/powerpc/include/asm/mpic_msgr.h index d4f471fb1031..088420d8aa59 100644 --- a/arch/powerpc/include/asm/mpic_msgr.h +++ b/arch/powerpc/include/asm/mpic_msgr.h @@ -122,9 +122,9 @@ static inline void mpic_msgr_set_destination(struct mpic_msgr *msgr, * @msgr: the message register whose IRQ is to be returned * * Returns the IRQ number associated with the given message register. - * NO_IRQ is returned if this message register is not capable of - * receiving interrupts. What message register can and cannot receive - * interrupts is specified in the device tree for the system. + * 0 is returned if this message register is not capable of receiving + * interrupts. What message register can and cannot receive interrupts is + * specified in the device tree for the system. */ static inline int mpic_msgr_get_irq(struct mpic_msgr *msgr) { diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h index 780847597514..c219ef7be53b 100644 --- a/arch/powerpc/include/asm/nohash/32/pgtable.h +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h @@ -267,7 +267,8 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, } -static inline void __ptep_set_access_flags(pte_t *ptep, pte_t entry) +static inline void __ptep_set_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h index d4d808cf905e..653a1838469d 100644 --- a/arch/powerpc/include/asm/nohash/64/pgtable.h +++ b/arch/powerpc/include/asm/nohash/64/pgtable.h @@ -300,7 +300,8 @@ static inline void pte_clear(struct mm_struct *mm, unsigned long addr, /* Set the dirty and/or accessed bits atomically in a linux PTE, this * function doesn't need to flush the hash entry */ -static inline void __ptep_set_access_flags(pte_t *ptep, pte_t entry) +static inline void __ptep_set_access_flags(struct mm_struct *mm, + pte_t *ptep, pte_t entry) { unsigned long bits = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); diff --git a/arch/powerpc/include/asm/parport.h b/arch/powerpc/include/asm/parport.h index a452968b29ea..6595ad1d18cc 100644 --- a/arch/powerpc/include/asm/parport.h +++ b/arch/powerpc/include/asm/parport.h @@ -28,7 +28,7 @@ static int parport_pc_find_nonpci_ports (int autoirq, int autodma) io1 = prop[1]; io2 = prop[2]; virq = irq_of_parse_and_map(np, 0); - if (virq == NO_IRQ) + if (!virq) continue; if (parport_pc_probe_port(io1, io2, virq, autodma, NULL, 0) diff --git a/arch/powerpc/include/asm/pnv-pci.h b/arch/powerpc/include/asm/pnv-pci.h index 1b46b52d3212..696438f09aea 100644 --- a/arch/powerpc/include/asm/pnv-pci.h +++ b/arch/powerpc/include/asm/pnv-pci.h @@ -16,7 +16,7 @@ #include <misc/cxl-base.h> #include <asm/opal-api.h> -#define PCI_SLOT_ID_PREFIX 0x8000000000000000 +#define PCI_SLOT_ID_PREFIX (1UL << 63) #define PCI_SLOT_ID(phb_id, bdfn) \ (PCI_SLOT_ID_PREFIX | ((uint64_t)(bdfn) << 16) | (phb_id)) @@ -63,6 +63,8 @@ struct pnv_php_slot { #define PNV_PHP_STATE_POPULATED 2 #define PNV_PHP_STATE_OFFLINE 3 int state; + int irq; + struct workqueue_struct *wq; struct device_node *dn; struct pci_dev *pdev; struct pci_bus *bus; diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h index 127ebf5862b4..54ff8ce7fa96 100644 --- a/arch/powerpc/include/asm/ppc-opcode.h +++ b/arch/powerpc/include/asm/ppc-opcode.h @@ -236,6 +236,7 @@ #define PPC_INST_STWU 0x94000000 #define PPC_INST_MFLR 0x7c0802a6 #define PPC_INST_MTLR 0x7c0803a6 +#define PPC_INST_MTCTR 0x7c0903a6 #define PPC_INST_CMPWI 0x2c000000 #define PPC_INST_CMPDI 0x2c200000 #define PPC_INST_CMPW 0x7c000000 @@ -250,6 +251,7 @@ #define PPC_INST_SUB 0x7c000050 #define PPC_INST_BLR 0x4e800020 #define PPC_INST_BLRL 0x4e800021 +#define PPC_INST_BCTR 0x4e800420 #define PPC_INST_MULLD 0x7c0001d2 #define PPC_INST_MULLW 0x7c0001d6 #define PPC_INST_MULHWU 0x7c000016 diff --git a/arch/powerpc/include/asm/ppc_asm.h b/arch/powerpc/include/asm/ppc_asm.h index d5d5b5e348f2..c73750b0d9fa 100644 --- a/arch/powerpc/include/asm/ppc_asm.h +++ b/arch/powerpc/include/asm/ppc_asm.h @@ -201,14 +201,12 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) #ifdef PPC64_ELF_ABI_v2 #define _GLOBAL(name) \ - .section ".text"; \ .align 2 ; \ .type name,@function; \ .globl name; \ name: #define _GLOBAL_TOC(name) \ - .section ".text"; \ .align 2 ; \ .type name,@function; \ .globl name; \ @@ -217,13 +215,6 @@ name: \ addi r2,r2,(.TOC.-0b)@l; \ .localentry name,.-name -#define _KPROBE(name) \ - .section ".kprobes.text","a"; \ - .align 2 ; \ - .type name,@function; \ - .globl name; \ -name: - #define DOTSYM(a) a #else @@ -232,35 +223,20 @@ name: #define GLUE(a,b) XGLUE(a,b) #define _GLOBAL(name) \ - .section ".text"; \ .align 2 ; \ .globl name; \ .globl GLUE(.,name); \ - .section ".opd","aw"; \ + .pushsection ".opd","aw"; \ name: \ .quad GLUE(.,name); \ .quad .TOC.@tocbase; \ .quad 0; \ - .previous; \ + .popsection; \ .type GLUE(.,name),@function; \ GLUE(.,name): #define _GLOBAL_TOC(name) _GLOBAL(name) -#define _KPROBE(name) \ - .section ".kprobes.text","a"; \ - .align 2 ; \ - .globl name; \ - .globl GLUE(.,name); \ - .section ".opd","aw"; \ -name: \ - .quad GLUE(.,name); \ - .quad .TOC.@tocbase; \ - .quad 0; \ - .previous; \ - .type GLUE(.,name),@function; \ -GLUE(.,name): - #define DOTSYM(a) GLUE(.,a) #endif @@ -272,20 +248,28 @@ GLUE(.,name): n: #define _GLOBAL(n) \ - .text; \ .stabs __stringify(n:F-1),N_FUN,0,0,n;\ .globl n; \ n: #define _GLOBAL_TOC(name) _GLOBAL(name) -#define _KPROBE(n) \ - .section ".kprobes.text","a"; \ - .globl n; \ -n: - #endif +/* + * __kprobes (the C annotation) puts the symbol into the .kprobes.text + * section, which gets emitted at the end of regular text. + * + * _ASM_NOKPROBE_SYMBOL and NOKPROBE_SYMBOL just adds the symbol to + * a blacklist. The former is for core kprobe functions/data, the + * latter is for those that incdentially must be excluded from probing + * and allows them to be linked at more optimal location within text. + */ +#define _ASM_NOKPROBE_SYMBOL(entry) \ + .pushsection "_kprobe_blacklist","aw"; \ + PPC_LONG (entry) ; \ + .popsection + #define FUNC_START(name) _GLOBAL(name) #define FUNC_END(name) @@ -527,7 +511,6 @@ END_FTR_SECTION_IFCLR(CPU_FTR_601) #endif #define MTMSRD(r) mtmsr r #define MTMSR_EERI(reg) mtmsr reg -#define CLR_TOP32(r) #endif #endif /* __KERNEL__ */ diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h index 68e3bf57b027..c07c31b0e89e 100644 --- a/arch/powerpc/include/asm/processor.h +++ b/arch/powerpc/include/asm/processor.h @@ -147,7 +147,7 @@ typedef struct { } mm_segment_t; #define TS_FPR(i) fp_state.fpr[i][TS_FPROFFSET] -#define TS_TRANS_FPR(i) transact_fp.fpr[i][TS_FPROFFSET] +#define TS_CKFPR(i) ckfp_state.fpr[i][TS_FPROFFSET] /* FP and VSX 0-31 register set */ struct thread_fp_state { @@ -257,6 +257,7 @@ struct thread_struct { int used_spe; /* set if process has used spe */ #endif /* CONFIG_SPE */ #ifdef CONFIG_PPC_TRANSACTIONAL_MEM + u8 load_tm; u64 tm_tfhar; /* Transaction fail handler addr */ u64 tm_texasr; /* Transaction exception & summary */ u64 tm_tfiar; /* Transaction fail instr address reg */ @@ -267,20 +268,17 @@ struct thread_struct { unsigned long tm_dscr; /* - * Transactional FP and VSX 0-31 register set. - * NOTE: the sense of these is the opposite of the integer ckpt_regs! + * Checkpointed FP and VSX 0-31 register set. * * When a transaction is active/signalled/scheduled etc., *regs is the * most recent set of/speculated GPRs with ckpt_regs being the older * checkpointed regs to which we roll back if transaction aborts. * - * However, fpr[] is the checkpointed 'base state' of FP regs, and - * transact_fpr[] is the new set of transactional values. - * VRs work the same way. + * These are analogous to how ckpt_regs and pt_regs work */ - struct thread_fp_state transact_fp; - struct thread_vr_state transact_vr; - unsigned long transact_vrsave; + struct thread_fp_state ckfp_state; /* Checkpointed FP state */ + struct thread_vr_state ckvr_state; /* Checkpointed VR state */ + unsigned long ckvrsave; /* Checkpointed VRSAVE */ #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ #ifdef CONFIG_KVM_BOOK3S_32_HANDLER void* kvm_shadow_vcpu; /* KVM internal data */ diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h index 978dada662ae..2a620789954b 100644 --- a/arch/powerpc/include/asm/reg.h +++ b/arch/powerpc/include/asm/reg.h @@ -475,6 +475,9 @@ #define HID0_POWER8_1TO4LPAR __MASK(51) #define HID0_POWER8_DYNLPARDIS __MASK(48) +/* POWER9 HID0 bits */ +#define HID0_POWER9_RADIX __MASK(63 - 8) + #define SPRN_HID1 0x3F1 /* Hardware Implementation Register 1 */ #ifdef CONFIG_6xx #define HID1_EMCP (1<<31) /* 7450 Machine Check Pin Enable */ @@ -1248,7 +1251,7 @@ static inline void mtmsr_isync(unsigned long val) : "memory") #endif -extern void msr_check_and_set(unsigned long bits); +extern unsigned long msr_check_and_set(unsigned long bits); extern bool strict_msr_control; extern void __msr_check_and_clear(unsigned long bits); static inline void msr_check_and_clear(unsigned long bits) diff --git a/arch/powerpc/include/asm/signal.h b/arch/powerpc/include/asm/signal.h index 9322c28aebd2..5ff77722a52d 100644 --- a/arch/powerpc/include/asm/signal.h +++ b/arch/powerpc/include/asm/signal.h @@ -5,6 +5,4 @@ #include <uapi/asm/signal.h> #include <uapi/asm/ptrace.h> -extern unsigned long get_tm_stackpointer(struct pt_regs *regs); - #endif /* _ASM_POWERPC_SIGNAL_H */ diff --git a/arch/powerpc/include/asm/tm.h b/arch/powerpc/include/asm/tm.h index c22d704b6d41..82e06ca3a49b 100644 --- a/arch/powerpc/include/asm/tm.h +++ b/arch/powerpc/include/asm/tm.h @@ -9,11 +9,6 @@ #ifndef __ASSEMBLY__ -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -extern void do_load_up_transact_fpu(struct thread_struct *thread); -extern void do_load_up_transact_altivec(struct thread_struct *thread); -#endif - extern void tm_enable(void); extern void tm_reclaim(struct thread_struct *thread, unsigned long orig_msr, uint8_t cause); diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index fe4c075bcf50..aded29ad2e8f 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -31,8 +31,7 @@ obj-y := cputable.o ptrace.o syscalls.o \ process.o systbl.o idle.o \ signal.o sysfs.o cacheinfo.o time.o \ prom.o traps.o setup-common.o \ - udbg.o misc.o io.o dma.o \ - misc_$(CONFIG_WORD_SIZE).o \ + udbg.o misc.o io.o dma.o misc_$(BITS).o \ of_platform.o prom_parse.o obj-$(CONFIG_PPC64) += setup_64.o sys_ppc32.o \ signal_64.o ptrace32.o \ @@ -70,23 +69,23 @@ obj-$(CONFIG_HIBERNATION) += swsusp.o suspend.o ifeq ($(CONFIG_FSL_BOOKE),y) obj-$(CONFIG_HIBERNATION) += swsusp_booke.o else -obj-$(CONFIG_HIBERNATION) += swsusp_$(CONFIG_WORD_SIZE).o +obj-$(CONFIG_HIBERNATION) += swsusp_$(BITS).o endif obj64-$(CONFIG_HIBERNATION) += swsusp_asm64.o -obj-$(CONFIG_MODULES) += module.o module_$(CONFIG_WORD_SIZE).o +obj-$(CONFIG_MODULES) += module.o module_$(BITS).o obj-$(CONFIG_44x) += cpu_setup_44x.o obj-$(CONFIG_PPC_FSL_BOOK3E) += cpu_setup_fsl_booke.o obj-$(CONFIG_PPC_DOORBELL) += dbell.o obj-$(CONFIG_JUMP_LABEL) += jump_label.o -extra-y := head_$(CONFIG_WORD_SIZE).o +extra-y := head_$(BITS).o extra-$(CONFIG_40x) := head_40x.o extra-$(CONFIG_44x) := head_44x.o extra-$(CONFIG_FSL_BOOKE) := head_fsl_booke.o extra-$(CONFIG_8xx) := head_8xx.o extra-y += vmlinux.lds -obj-$(CONFIG_RELOCATABLE) += reloc_$(CONFIG_WORD_SIZE).o +obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o obj-$(CONFIG_PPC32) += entry_32.o setup_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o @@ -104,11 +103,11 @@ obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-$(CONFIG_SWIOTLB) += dma-swiotlb.o pci64-$(CONFIG_PPC64) += pci_dn.o pci-hotplug.o isa-bridge.o -obj-$(CONFIG_PCI) += pci_$(CONFIG_WORD_SIZE).o $(pci64-y) \ +obj-$(CONFIG_PCI) += pci_$(BITS).o $(pci64-y) \ pci-common.o pci_of_scan.o obj-$(CONFIG_PCI_MSI) += msi.o obj-$(CONFIG_KEXEC) += machine_kexec.o crash.o \ - machine_kexec_$(CONFIG_WORD_SIZE).o + machine_kexec_$(BITS).o obj-$(CONFIG_AUDIT) += audit.o obj64-$(CONFIG_AUDIT) += compat_audit.o diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index a51ae9b165e0..caec7bf3b99a 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -142,12 +142,12 @@ int main(void) DEFINE(THREAD_TM_PPR, offsetof(struct thread_struct, tm_ppr)); DEFINE(THREAD_TM_DSCR, offsetof(struct thread_struct, tm_dscr)); DEFINE(PT_CKPT_REGS, offsetof(struct thread_struct, ckpt_regs)); - DEFINE(THREAD_TRANSACT_VRSTATE, offsetof(struct thread_struct, - transact_vr)); - DEFINE(THREAD_TRANSACT_VRSAVE, offsetof(struct thread_struct, - transact_vrsave)); - DEFINE(THREAD_TRANSACT_FPSTATE, offsetof(struct thread_struct, - transact_fp)); + DEFINE(THREAD_CKVRSTATE, offsetof(struct thread_struct, + ckvr_state)); + DEFINE(THREAD_CKVRSAVE, offsetof(struct thread_struct, + ckvrsave)); + DEFINE(THREAD_CKFPSTATE, offsetof(struct thread_struct, + ckfp_state)); /* Local pt_regs on stack for Transactional Memory funcs. */ DEFINE(TM_FRAME_SIZE, STACK_FRAME_OVERHEAD + sizeof(struct pt_regs) + 16); diff --git a/arch/powerpc/kernel/cputable.c b/arch/powerpc/kernel/cputable.c index 74248ab18e98..6c4646ac9234 100644 --- a/arch/powerpc/kernel/cputable.c +++ b/arch/powerpc/kernel/cputable.c @@ -506,6 +506,25 @@ static struct cpu_spec __initdata cpu_specs[] = { .machine_check_early = __machine_check_early_realmode_p8, .platform = "power8", }, + { /* Power9 DD1*/ + .pvr_mask = 0xffffff00, + .pvr_value = 0x004e0100, + .cpu_name = "POWER9 (raw)", + .cpu_features = CPU_FTRS_POWER9_DD1, + .cpu_user_features = COMMON_USER_POWER9, + .cpu_user_features2 = COMMON_USER2_POWER9, + .mmu_features = MMU_FTRS_POWER9, + .icache_bsize = 128, + .dcache_bsize = 128, + .num_pmcs = 6, + .pmc_type = PPC_PMC_IBM, + .oprofile_cpu_type = "ppc64/power9", + .oprofile_type = PPC_OPROFILE_INVALID, + .cpu_setup = __setup_cpu_power9, + .cpu_restore = __restore_cpu_power9, + .flush_tlb = __flush_tlb_power9, + .platform = "power9", + }, { /* Power9 */ .pvr_mask = 0xffff0000, .pvr_value = 0x004e0000, diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c index 7429556eb8df..f25731627d7f 100644 --- a/arch/powerpc/kernel/eeh.c +++ b/arch/powerpc/kernel/eeh.c @@ -116,6 +116,7 @@ struct eeh_ops *eeh_ops = NULL; /* Lock to avoid races due to multiple reports of an error */ DEFINE_RAW_SPINLOCK(confirm_error_lock); +EXPORT_SYMBOL_GPL(confirm_error_lock); /* Lock to protect passed flags */ static DEFINE_MUTEX(eeh_dev_mutex); @@ -1044,7 +1045,7 @@ int eeh_init(void) if (eeh_enabled()) pr_info("EEH: PCI Enhanced I/O Error Handling Enabled\n"); else - pr_warn("EEH: No capable adapters found\n"); + pr_info("EEH: No capable adapters found\n"); return ret; } @@ -1502,6 +1503,7 @@ int eeh_pe_set_option(struct eeh_pe *pe, int option) break; case EEH_OPT_THAW_MMIO: case EEH_OPT_THAW_DMA: + case EEH_OPT_FREEZE_PE: if (!eeh_ops || !eeh_ops->set_option) { ret = -ENOENT; break; diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c index 5f36e8a70daa..a62be72da274 100644 --- a/arch/powerpc/kernel/eeh_driver.c +++ b/arch/powerpc/kernel/eeh_driver.c @@ -993,9 +993,17 @@ static void eeh_handle_special_event(void) /* Notify all devices to be down */ eeh_pe_state_clear(pe, EEH_PE_PRI_BUS); - bus = eeh_pe_bus_get(phb_pe); eeh_pe_dev_traverse(pe, eeh_report_failure, NULL); + bus = eeh_pe_bus_get(phb_pe); + if (!bus) { + pr_err("%s: Cannot find PCI bus for " + "PHB#%d-PE#%x\n", + __func__, + pe->phb->global_number, + pe->addr); + break; + } pci_hp_remove_devices(bus); } pci_unlock_rescan_remove(); diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c index f0520da85759..de7d091c4c31 100644 --- a/arch/powerpc/kernel/eeh_pe.c +++ b/arch/powerpc/kernel/eeh_pe.c @@ -581,6 +581,7 @@ void eeh_pe_state_mark(struct eeh_pe *pe, int state) { eeh_pe_traverse(pe, __eeh_pe_state_mark, &state); } +EXPORT_SYMBOL_GPL(eeh_pe_state_mark); static void *__eeh_pe_dev_mode_mark(void *data, void *flag) { diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S index 9899032230b4..83428a283fa0 100644 --- a/arch/powerpc/kernel/entry_32.S +++ b/arch/powerpc/kernel/entry_32.S @@ -654,7 +654,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_SPE) #endif /* CONFIG_SMP */ tophys(r0,r4) - CLR_TOP32(r0) mtspr SPRN_SPRG_THREAD,r0 /* Update current THREAD phys addr */ lwz r1,KSP(r4) /* Load new stack pointer */ diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S index 5afd03e5e8b8..51df82b61084 100644 --- a/arch/powerpc/kernel/entry_64.S +++ b/arch/powerpc/kernel/entry_64.S @@ -139,7 +139,7 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) #ifdef CONFIG_PPC_BOOK3E wrteei 1 #else - ld r11,PACAKMSR(r13) + li r11,MSR_RI ori r11,r11,MSR_EE mtmsrd r11,1 #endif /* CONFIG_PPC_BOOK3E */ @@ -195,7 +195,6 @@ system_call: /* label this so stack traces look sane */ #ifdef CONFIG_PPC_BOOK3E wrteei 0 #else - ld r10,PACAKMSR(r13) /* * For performance reasons we clear RI the same time that we * clear EE. We only need to clear RI just before we restore r13 @@ -203,8 +202,7 @@ system_call: /* label this so stack traces look sane */ * We have to be careful to restore RI if we branch anywhere from * here (eg syscall_exit_work). */ - li r9,MSR_RI - andc r11,r10,r9 + li r11,0 mtmsrd r11,1 #endif /* CONFIG_PPC_BOOK3E */ @@ -221,13 +219,12 @@ system_call: /* label this so stack traces look sane */ #endif 2: addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_PPC_BOOK3S + li r10,MSR_RI mtmsrd r10,1 /* Restore RI */ #endif bl restore_math #ifdef CONFIG_PPC_BOOK3S - ld r10,PACAKMSR(r13) - li r9,MSR_RI - andc r11,r10,r9 /* Re-clear RI */ + li r11,0 mtmsrd r11,1 #endif ld r8,_MSR(r1) @@ -308,6 +305,7 @@ syscall_enosys: syscall_exit_work: #ifdef CONFIG_PPC_BOOK3S + li r10,MSR_RI mtmsrd r10,1 /* Restore RI */ #endif /* If TIF_RESTOREALL is set, don't scribble on either r3 or ccr. @@ -354,7 +352,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR) #ifdef CONFIG_PPC_BOOK3E wrteei 1 #else - ld r10,PACAKMSR(r13) + li r10,MSR_RI ori r10,r10,MSR_EE mtmsrd r10,1 #endif /* CONFIG_PPC_BOOK3E */ @@ -619,7 +617,7 @@ _GLOBAL(ret_from_except_lite) #ifdef CONFIG_PPC_BOOK3E wrteei 0 #else - ld r10,PACAKMSR(r13) /* Get kernel MSR without EE */ + li r10,MSR_RI mtmsrd r10,1 /* Update machine state */ #endif /* CONFIG_PPC_BOOK3E */ @@ -751,7 +749,7 @@ resume_kernel: #ifdef CONFIG_PPC_BOOK3E wrteei 0 #else - ld r10,PACAKMSR(r13) /* Get kernel MSR without EE */ + li r10,MSR_RI mtmsrd r10,1 /* Update machine state */ #endif /* CONFIG_PPC_BOOK3E */ #endif /* CONFIG_PREEMPT */ @@ -841,8 +839,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR) * userspace and we take an exception after restoring r13, * we end up corrupting the userspace r13 value. */ - ld r4,PACAKMSR(r13) /* Get kernel MSR without EE */ - andc r4,r4,r0 /* r0 contains MSR_RI here */ + li r4,0 mtmsrd r4,1 #ifdef CONFIG_PPC_TRANSACTIONAL_MEM diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index bffec73dbffc..08992f8f5036 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -16,72 +16,71 @@ #include <asm/exception-64s.h> #include <asm/ptrace.h> #include <asm/cpuidle.h> +#include <asm/head-64.h> /* + * There are a few constraints to be concerned with. + * - Real mode exceptions code/data must be located at their physical location. + * - Virtual mode exceptions must be mapped at their 0xc000... location. + * - Fixed location code must not call directly beyond the __end_interrupts + * area when built with CONFIG_RELOCATABLE. LOAD_HANDLER / bctr sequence + * must be used. + * - LOAD_HANDLER targets must be within first 64K of physical 0 / + * virtual 0xc00... + * - Conditional branch targets must be within +/-32K of caller. + * + * "Virtual exceptions" run with relocation on (MSR_IR=1, MSR_DR=1), and + * therefore don't have to run in physically located code or rfid to + * virtual mode kernel code. However on relocatable kernels they do have + * to branch to KERNELBASE offset because the rest of the kernel (outside + * the exception vectors) may be located elsewhere. + * + * Virtual exceptions correspond with physical, except their entry points + * are offset by 0xc000000000000000 and also tend to get an added 0x4000 + * offset applied. Virtual exceptions are enabled with the Alternate + * Interrupt Location (AIL) bit set in the LPCR. However this does not + * guarantee they will be delivered virtually. Some conditions (see the ISA) + * cause exceptions to be delivered in real mode. + * + * It's impossible to receive interrupts below 0x300 via AIL. + * + * KVM: None of the virtual exceptions are from the guest. Anything that + * escalated to HV=1 from HV=0 is delivered via real mode handlers. + * + * * We layout physical memory as follows: * 0x0000 - 0x00ff : Secondary processor spin code - * 0x0100 - 0x17ff : pSeries Interrupt prologs - * 0x1800 - 0x4000 : interrupt support common interrupt prologs - * 0x4000 - 0x5fff : pSeries interrupts with IR=1,DR=1 - * 0x6000 - 0x6fff : more interrupt support including for IR=1,DR=1 + * 0x0100 - 0x18ff : Real mode pSeries interrupt vectors + * 0x1900 - 0x3fff : Real mode trampolines + * 0x4000 - 0x58ff : Relon (IR=1,DR=1) mode pSeries interrupt vectors + * 0x5900 - 0x6fff : Relon mode trampolines * 0x7000 - 0x7fff : FWNMI data area - * 0x8000 - 0x8fff : Initial (CPU0) segment table - * 0x9000 - : Early init and support code + * 0x8000 - .... : Common interrupt handlers, remaining early + * setup code, rest of kernel. + * + * We could reclaim 0x4000-0x42ff for real mode trampolines if the space + * is necessary. Until then it's more consistent to explicitly put VIRT_NONE + * vectors there. */ - /* Syscall routine is used twice, in reloc-off and reloc-on paths */ -#define SYSCALL_PSERIES_1 \ -BEGIN_FTR_SECTION \ - cmpdi r0,0x1ebe ; \ - beq- 1f ; \ -END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) \ - mr r9,r13 ; \ - GET_PACA(r13) ; \ - mfspr r11,SPRN_SRR0 ; \ -0: - -#define SYSCALL_PSERIES_2_RFID \ - mfspr r12,SPRN_SRR1 ; \ - ld r10,PACAKBASE(r13) ; \ - LOAD_HANDLER(r10, system_call_entry) ; \ - mtspr SPRN_SRR0,r10 ; \ - ld r10,PACAKMSR(r13) ; \ - mtspr SPRN_SRR1,r10 ; \ - rfid ; \ - b . ; /* prevent speculative execution */ - -#define SYSCALL_PSERIES_3 \ - /* Fast LE/BE switch system call */ \ -1: mfspr r12,SPRN_SRR1 ; \ - xori r12,r12,MSR_LE ; \ - mtspr SPRN_SRR1,r12 ; \ - rfid ; /* return to userspace */ \ - b . ; /* prevent speculative execution */ - -#if defined(CONFIG_RELOCATABLE) - /* - * We can't branch directly so we do it via the CTR which - * is volatile across system calls. - */ -#define SYSCALL_PSERIES_2_DIRECT \ - mflr r10 ; \ - ld r12,PACAKBASE(r13) ; \ - LOAD_HANDLER(r12, system_call_entry) ; \ - mtctr r12 ; \ - mfspr r12,SPRN_SRR1 ; \ - /* Re-use of r13... No spare regs to do this */ \ - li r13,MSR_RI ; \ - mtmsrd r13,1 ; \ - GET_PACA(r13) ; /* get r13 back */ \ - bctr ; +OPEN_FIXED_SECTION(real_vectors, 0x0100, 0x1900) +OPEN_FIXED_SECTION(real_trampolines, 0x1900, 0x4000) +OPEN_FIXED_SECTION(virt_vectors, 0x4000, 0x5900) +OPEN_FIXED_SECTION(virt_trampolines, 0x5900, 0x7000) +#if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) +/* + * Data area reserved for FWNMI option. + * This address (0x7000) is fixed by the RPA. + * pseries and powernv need to keep the whole page from + * 0x7000 to 0x8000 free for use by the firmware + */ +ZERO_FIXED_SECTION(fwnmi_page, 0x7000, 0x8000) +OPEN_TEXT_SECTION(0x8000) #else - /* We can branch directly */ -#define SYSCALL_PSERIES_2_DIRECT \ - mfspr r12,SPRN_SRR1 ; \ - li r10,MSR_RI ; \ - mtmsrd r10,1 ; /* Set RI (EE=0) */ \ - b system_call_common ; +OPEN_TEXT_SECTION(0x7000) #endif +USE_FIXED_SECTION(real_vectors) + /* * This is the start of the interrupt handlers for pSeries * This code runs with relocation off. @@ -90,12 +89,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) \ * Therefore any relative branches in this section must only * branch to labels in this section. */ - . = 0x100 .globl __start_interrupts __start_interrupts: - .globl system_reset_pSeries; -system_reset_pSeries: +/* No virt vectors corresponding with 0x0..0x100 */ +EXC_VIRT_NONE(0x4000, 0x4100) + +EXC_REAL_BEGIN(system_reset, 0x100, 0x200) SET_SCRATCH0(r13) #ifdef CONFIG_PPC_P7_NAP BEGIN_FTR_SECTION @@ -136,9 +136,22 @@ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) #endif /* CONFIG_PPC_P7_NAP */ EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, system_reset_common, EXC_STD, NOTEST, 0x100) +EXC_REAL_END(system_reset, 0x100, 0x200) +EXC_VIRT_NONE(0x4100, 0x4200) +EXC_COMMON(system_reset_common, 0x100, system_reset_exception) + +#ifdef CONFIG_PPC_PSERIES +/* + * Vectors for the FWNMI option. Share common code. + */ +TRAMP_REAL_BEGIN(system_reset_fwnmi) + SET_SCRATCH0(r13) /* save r13 */ + EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, system_reset_common, EXC_STD, + NOTEST, 0x100) +#endif /* CONFIG_PPC_PSERIES */ + - . = 0x200 -machine_check_pSeries_1: +EXC_REAL_BEGIN(machine_check, 0x200, 0x300) /* This is moved out of line as it can be patched by FW, but * some code path might still want to branch into the original * vector @@ -158,253 +171,9 @@ BEGIN_FTR_SECTION FTR_SECTION_ELSE b machine_check_pSeries_0 ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE) - - . = 0x300 - .globl data_access_pSeries -data_access_pSeries: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, data_access_common, EXC_STD, - KVMTEST, 0x300) - - . = 0x380 - .globl data_access_slb_pSeries -data_access_slb_pSeries: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST, 0x380) - std r3,PACA_EXSLB+EX_R3(r13) - mfspr r3,SPRN_DAR - mfspr r12,SPRN_SRR1 -#ifndef CONFIG_RELOCATABLE - b slb_miss_realmode -#else - /* - * We can't just use a direct branch to slb_miss_realmode - * because the distance from here to there depends on where - * the kernel ends up being put. - */ - mfctr r11 - ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, slb_miss_realmode) - mtctr r10 - bctr -#endif - - STD_EXCEPTION_PSERIES(0x400, instruction_access) - - . = 0x480 - .globl instruction_access_slb_pSeries -instruction_access_slb_pSeries: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST, 0x480) - std r3,PACA_EXSLB+EX_R3(r13) - mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ - mfspr r12,SPRN_SRR1 -#ifndef CONFIG_RELOCATABLE - b slb_miss_realmode -#else - mfctr r11 - ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, slb_miss_realmode) - mtctr r10 - bctr -#endif - - /* We open code these as we can't have a ". = x" (even with - * x = "." within a feature section - */ - . = 0x500; - .globl hardware_interrupt_pSeries; - .globl hardware_interrupt_hv; -hardware_interrupt_pSeries: -hardware_interrupt_hv: - BEGIN_FTR_SECTION - _MASKABLE_EXCEPTION_PSERIES(0x502, hardware_interrupt, - EXC_HV, SOFTEN_TEST_HV) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0x502) - FTR_SECTION_ELSE - _MASKABLE_EXCEPTION_PSERIES(0x500, hardware_interrupt, - EXC_STD, SOFTEN_TEST_PR) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x500) - ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) - - STD_EXCEPTION_PSERIES(0x600, alignment) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x600) - - STD_EXCEPTION_PSERIES(0x700, program_check) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x700) - - STD_EXCEPTION_PSERIES(0x800, fp_unavailable) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x800) - - . = 0x900 - .globl decrementer_pSeries -decrementer_pSeries: - _MASKABLE_EXCEPTION_PSERIES(0x900, decrementer, EXC_STD, SOFTEN_TEST_PR) - - STD_EXCEPTION_HV(0x980, 0x982, hdecrementer) - - MASKABLE_EXCEPTION_PSERIES(0xa00, 0xa00, doorbell_super) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xa00) - - STD_EXCEPTION_PSERIES(0xb00, trap_0b) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xb00) - - . = 0xc00 - .globl system_call_pSeries -system_call_pSeries: - /* - * If CONFIG_KVM_BOOK3S_64_HANDLER is set, save the PPR (on systems - * that support it) before changing to HMT_MEDIUM. That allows the KVM - * code to save that value into the guest state (it is the guest's PPR - * value). Otherwise just change to HMT_MEDIUM as userspace has - * already saved the PPR. - */ -#ifdef CONFIG_KVM_BOOK3S_64_HANDLER - SET_SCRATCH0(r13) - GET_PACA(r13) - std r9,PACA_EXGEN+EX_R9(r13) - OPT_GET_SPR(r9, SPRN_PPR, CPU_FTR_HAS_PPR); - HMT_MEDIUM; - std r10,PACA_EXGEN+EX_R10(r13) - OPT_SAVE_REG_TO_PACA(PACA_EXGEN+EX_PPR, r9, CPU_FTR_HAS_PPR); - mfcr r9 - KVMTEST(0xc00) - GET_SCRATCH0(r13) -#else - HMT_MEDIUM; -#endif - SYSCALL_PSERIES_1 - SYSCALL_PSERIES_2_RFID - SYSCALL_PSERIES_3 - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xc00) - - STD_EXCEPTION_PSERIES(0xd00, single_step) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xd00) - - /* At 0xe??? we have a bunch of hypervisor exceptions, we branch - * out of line to handle them - */ - . = 0xe00 -hv_data_storage_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_data_storage_hv - - . = 0xe20 -hv_instr_storage_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_instr_storage_hv - - . = 0xe40 -emulation_assist_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b emulation_assist_hv - - . = 0xe60 -hv_exception_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b hmi_exception_early - - . = 0xe80 -hv_doorbell_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_doorbell_hv - - . = 0xea0 -hv_virt_irq_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_virt_irq_hv - - /* We need to deal with the Altivec unavailable exception - * here which is at 0xf20, thus in the middle of the - * prolog code of the PerformanceMonitor one. A little - * trickery is thus necessary - */ - . = 0xf00 -performance_monitor_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b performance_monitor_pSeries - - . = 0xf20 -altivec_unavailable_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b altivec_unavailable_pSeries - - . = 0xf40 -vsx_unavailable_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b vsx_unavailable_pSeries - - . = 0xf60 -facility_unavailable_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b facility_unavailable_pSeries - - . = 0xf80 -hv_facility_unavailable_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b facility_unavailable_hv - -#ifdef CONFIG_CBE_RAS - STD_EXCEPTION_HV(0x1200, 0x1202, cbe_system_error) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_HV, 0x1202) -#endif /* CONFIG_CBE_RAS */ - - STD_EXCEPTION_PSERIES(0x1300, instruction_breakpoint) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_STD, 0x1300) - - . = 0x1500 - .global denorm_exception_hv -denorm_exception_hv: - mtspr SPRN_SPRG_HSCRATCH0,r13 - EXCEPTION_PROLOG_0(PACA_EXGEN) - EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x1500) - -#ifdef CONFIG_PPC_DENORMALISATION - mfspr r10,SPRN_HSRR1 - mfspr r11,SPRN_HSRR0 /* save HSRR0 */ - andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */ - addi r11,r11,-4 /* HSRR0 is next instruction */ - bne+ denorm_assist -#endif - - KVMTEST(0x1500) - EXCEPTION_PROLOG_PSERIES_1(denorm_common, EXC_HV) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_STD, 0x1500) - -#ifdef CONFIG_CBE_RAS - STD_EXCEPTION_HV(0x1600, 0x1602, cbe_maintenance) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_HV, 0x1602) -#endif /* CONFIG_CBE_RAS */ - - STD_EXCEPTION_PSERIES(0x1700, altivec_assist) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x1700) - -#ifdef CONFIG_CBE_RAS - STD_EXCEPTION_HV(0x1800, 0x1802, cbe_thermal) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_HV, 0x1802) -#else - . = 0x1800 -#endif /* CONFIG_CBE_RAS */ - - -/*** Out of line interrupts support ***/ - - .align 7 - /* moved from 0x200 */ -machine_check_powernv_early: +EXC_REAL_END(machine_check, 0x200, 0x300) +EXC_VIRT_NONE(0x4200, 0x4300) +TRAMP_REAL_BEGIN(machine_check_powernv_early) BEGIN_FTR_SECTION EXCEPTION_PROLOG_1(PACA_EXMC, NOTEST, 0x200) /* @@ -457,7 +226,6 @@ BEGIN_FTR_SECTION mfmsr r11 /* get MSR value */ ori r11,r11,MSR_ME /* turn on ME bit */ ori r11,r11,MSR_RI /* turn on RI bit */ - ld r12,PACAKBASE(r13) /* get high part of &label */ LOAD_HANDLER(r12, machine_check_handle_early) 1: mtspr SPRN_SRR0,r12 mtspr SPRN_SRR1,r11 @@ -470,7 +238,6 @@ BEGIN_FTR_SECTION */ addi r1,r1,INT_FRAME_SIZE /* go back to previous stack frame */ ld r11,PACAKMSR(r13) - ld r12,PACAKBASE(r13) LOAD_HANDLER(r12, unrecover_mce) li r10,MSR_ME andc r11,r11,r10 /* Turn off MSR_ME */ @@ -478,20 +245,19 @@ BEGIN_FTR_SECTION b . /* prevent speculative execution */ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE) -machine_check_pSeries: +TRAMP_REAL_BEGIN(machine_check_pSeries) .globl machine_check_fwnmi machine_check_fwnmi: SET_SCRATCH0(r13) /* save r13 */ EXCEPTION_PROLOG_0(PACA_EXMC) machine_check_pSeries_0: - EXCEPTION_PROLOG_1(PACA_EXMC, KVMTEST, 0x200) + EXCEPTION_PROLOG_1(PACA_EXMC, KVMTEST_PR, 0x200) /* * The following is essentially EXCEPTION_PROLOG_PSERIES_1 with the * difference that MSR_RI is not enabled, because PACA_EXMC is being * used, so nested machine check corrupts it. machine_check_common * enables MSR_RI. */ - ld r12,PACAKBASE(r13) ld r10,PACAKMSR(r13) xori r10,r10,MSR_RI mfspr r11,SPRN_SRR0 @@ -502,488 +268,13 @@ machine_check_pSeries_0: rfid b . /* prevent speculative execution */ - KVM_HANDLER_SKIP(PACA_EXMC, EXC_STD, 0x200) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_STD, 0x300) - KVM_HANDLER_SKIP(PACA_EXSLB, EXC_STD, 0x380) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x400) - KVM_HANDLER(PACA_EXSLB, EXC_STD, 0x480) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x900) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0x982) - -#ifdef CONFIG_PPC_DENORMALISATION -denorm_assist: -BEGIN_FTR_SECTION -/* - * To denormalise we need to move a copy of the register to itself. - * For POWER6 do that here for all FP regs. - */ - mfmsr r10 - ori r10,r10,(MSR_FP|MSR_FE0|MSR_FE1) - xori r10,r10,(MSR_FE0|MSR_FE1) - mtmsrd r10 - sync - -#define FMR2(n) fmr (n), (n) ; fmr n+1, n+1 -#define FMR4(n) FMR2(n) ; FMR2(n+2) -#define FMR8(n) FMR4(n) ; FMR4(n+4) -#define FMR16(n) FMR8(n) ; FMR8(n+8) -#define FMR32(n) FMR16(n) ; FMR16(n+16) - FMR32(0) - -FTR_SECTION_ELSE -/* - * To denormalise we need to move a copy of the register to itself. - * For POWER7 do that here for the first 32 VSX registers only. - */ - mfmsr r10 - oris r10,r10,MSR_VSX@h - mtmsrd r10 - sync - -#define XVCPSGNDP2(n) XVCPSGNDP(n,n,n) ; XVCPSGNDP(n+1,n+1,n+1) -#define XVCPSGNDP4(n) XVCPSGNDP2(n) ; XVCPSGNDP2(n+2) -#define XVCPSGNDP8(n) XVCPSGNDP4(n) ; XVCPSGNDP4(n+4) -#define XVCPSGNDP16(n) XVCPSGNDP8(n) ; XVCPSGNDP8(n+8) -#define XVCPSGNDP32(n) XVCPSGNDP16(n) ; XVCPSGNDP16(n+16) - XVCPSGNDP32(0) - -ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_206) - -BEGIN_FTR_SECTION - b denorm_done -END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S) -/* - * To denormalise we need to move a copy of the register to itself. - * For POWER8 we need to do that for all 64 VSX registers - */ - XVCPSGNDP32(32) -denorm_done: - mtspr SPRN_HSRR0,r11 - mtcrf 0x80,r9 - ld r9,PACA_EXGEN+EX_R9(r13) - RESTORE_PPR_PACA(PACA_EXGEN, r10) -BEGIN_FTR_SECTION - ld r10,PACA_EXGEN+EX_CFAR(r13) - mtspr SPRN_CFAR,r10 -END_FTR_SECTION_IFSET(CPU_FTR_CFAR) - ld r10,PACA_EXGEN+EX_R10(r13) - ld r11,PACA_EXGEN+EX_R11(r13) - ld r12,PACA_EXGEN+EX_R12(r13) - ld r13,PACA_EXGEN+EX_R13(r13) - HRFID - b . -#endif - - .align 7 - /* moved from 0xe00 */ - STD_EXCEPTION_HV_OOL(0xe02, h_data_storage) - KVM_HANDLER_SKIP(PACA_EXGEN, EXC_HV, 0xe02) - STD_EXCEPTION_HV_OOL(0xe22, h_instr_storage) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xe22) - STD_EXCEPTION_HV_OOL(0xe42, emulation_assist) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xe42) - MASKABLE_EXCEPTION_HV_OOL(0xe62, hmi_exception) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xe62) - - MASKABLE_EXCEPTION_HV_OOL(0xe82, h_doorbell) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xe82) - - MASKABLE_EXCEPTION_HV_OOL(0xea2, h_virt_irq) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xea2) - - /* moved from 0xf00 */ - STD_EXCEPTION_PSERIES_OOL(0xf00, performance_monitor) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xf00) - STD_EXCEPTION_PSERIES_OOL(0xf20, altivec_unavailable) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xf20) - STD_EXCEPTION_PSERIES_OOL(0xf40, vsx_unavailable) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xf40) - STD_EXCEPTION_PSERIES_OOL(0xf60, facility_unavailable) - KVM_HANDLER(PACA_EXGEN, EXC_STD, 0xf60) - STD_EXCEPTION_HV_OOL(0xf82, facility_unavailable) - KVM_HANDLER(PACA_EXGEN, EXC_HV, 0xf82) - -/* - * An interrupt came in while soft-disabled. We set paca->irq_happened, then: - * - If it was a decrementer interrupt, we bump the dec to max and and return. - * - If it was a doorbell we return immediately since doorbells are edge - * triggered and won't automatically refire. - * - If it was a HMI we return immediately since we handled it in realmode - * and it won't refire. - * - else we hard disable and return. - * This is called with r10 containing the value to OR to the paca field. - */ -#define MASKED_INTERRUPT(_H) \ -masked_##_H##interrupt: \ - std r11,PACA_EXGEN+EX_R11(r13); \ - lbz r11,PACAIRQHAPPENED(r13); \ - or r11,r11,r10; \ - stb r11,PACAIRQHAPPENED(r13); \ - cmpwi r10,PACA_IRQ_DEC; \ - bne 1f; \ - lis r10,0x7fff; \ - ori r10,r10,0xffff; \ - mtspr SPRN_DEC,r10; \ - b 2f; \ -1: cmpwi r10,PACA_IRQ_DBELL; \ - beq 2f; \ - cmpwi r10,PACA_IRQ_HMI; \ - beq 2f; \ - mfspr r10,SPRN_##_H##SRR1; \ - rldicl r10,r10,48,1; /* clear MSR_EE */ \ - rotldi r10,r10,16; \ - mtspr SPRN_##_H##SRR1,r10; \ -2: mtcrf 0x80,r9; \ - ld r9,PACA_EXGEN+EX_R9(r13); \ - ld r10,PACA_EXGEN+EX_R10(r13); \ - ld r11,PACA_EXGEN+EX_R11(r13); \ - GET_SCRATCH0(r13); \ - ##_H##rfid; \ - b . - - MASKED_INTERRUPT() - MASKED_INTERRUPT(H) - -/* - * Called from arch_local_irq_enable when an interrupt needs - * to be resent. r3 contains 0x500, 0x900, 0xa00 or 0xe80 to indicate - * which kind of interrupt. MSR:EE is already off. We generate a - * stackframe like if a real interrupt had happened. - * - * Note: While MSR:EE is off, we need to make sure that _MSR - * in the generated frame has EE set to 1 or the exception - * handler will not properly re-enable them. - */ -_GLOBAL(__replay_interrupt) - /* We are going to jump to the exception common code which - * will retrieve various register values from the PACA which - * we don't give a damn about, so we don't bother storing them. - */ - mfmsr r12 - mflr r11 - mfcr r9 - ori r12,r12,MSR_EE - cmpwi r3,0x900 - beq decrementer_common - cmpwi r3,0x500 - beq hardware_interrupt_common -BEGIN_FTR_SECTION - cmpwi r3,0xe80 - beq h_doorbell_common - cmpwi r3,0xea0 - beq h_virt_irq_common - cmpwi r3,0xe60 - beq hmi_exception_common -FTR_SECTION_ELSE - cmpwi r3,0xa00 - beq doorbell_super_common -ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE) - blr - -#ifdef CONFIG_PPC_PSERIES -/* - * Vectors for the FWNMI option. Share common code. - */ - .globl system_reset_fwnmi - .align 7 -system_reset_fwnmi: - SET_SCRATCH0(r13) /* save r13 */ - EXCEPTION_PROLOG_PSERIES(PACA_EXGEN, system_reset_common, EXC_STD, - NOTEST, 0x100) - -#endif /* CONFIG_PPC_PSERIES */ - -#ifdef CONFIG_KVM_BOOK3S_64_HANDLER -kvmppc_skip_interrupt: - /* - * Here all GPRs are unchanged from when the interrupt happened - * except for r13, which is saved in SPRG_SCRATCH0. - */ - mfspr r13, SPRN_SRR0 - addi r13, r13, 4 - mtspr SPRN_SRR0, r13 - GET_SCRATCH0(r13) - rfid - b . - -kvmppc_skip_Hinterrupt: - /* - * Here all GPRs are unchanged from when the interrupt happened - * except for r13, which is saved in SPRG_SCRATCH0. - */ - mfspr r13, SPRN_HSRR0 - addi r13, r13, 4 - mtspr SPRN_HSRR0, r13 - GET_SCRATCH0(r13) - hrfid - b . -#endif - -/* - * Ensure that any handlers that get invoked from the exception prologs - * above are below the first 64KB (0x10000) of the kernel image because - * the prologs assemble the addresses of these handlers using the - * LOAD_HANDLER macro, which uses an ori instruction. - */ - -/*** Common interrupt handlers ***/ - - STD_EXCEPTION_COMMON(0x100, system_reset, system_reset_exception) - - STD_EXCEPTION_COMMON_ASYNC(0x500, hardware_interrupt, do_IRQ) - STD_EXCEPTION_COMMON_ASYNC(0x900, decrementer, timer_interrupt) - STD_EXCEPTION_COMMON(0x980, hdecrementer, hdec_interrupt) -#ifdef CONFIG_PPC_DOORBELL - STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, doorbell_exception) -#else - STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, unknown_exception) -#endif - STD_EXCEPTION_COMMON(0xb00, trap_0b, unknown_exception) - STD_EXCEPTION_COMMON(0xd00, single_step, single_step_exception) - STD_EXCEPTION_COMMON(0xe00, trap_0e, unknown_exception) - STD_EXCEPTION_COMMON(0xe40, emulation_assist, emulation_assist_interrupt) - STD_EXCEPTION_COMMON_ASYNC(0xe60, hmi_exception, handle_hmi_exception) -#ifdef CONFIG_PPC_DOORBELL - STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, doorbell_exception) -#else - STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, unknown_exception) -#endif - STD_EXCEPTION_COMMON_ASYNC(0xea0, h_virt_irq, do_IRQ) - STD_EXCEPTION_COMMON_ASYNC(0xf00, performance_monitor, performance_monitor_exception) - STD_EXCEPTION_COMMON(0x1300, instruction_breakpoint, instruction_breakpoint_exception) - STD_EXCEPTION_COMMON(0x1502, denorm, unknown_exception) -#ifdef CONFIG_ALTIVEC - STD_EXCEPTION_COMMON(0x1700, altivec_assist, altivec_assist_exception) -#else - STD_EXCEPTION_COMMON(0x1700, altivec_assist, unknown_exception) -#endif - - /* - * Relocation-on interrupts: A subset of the interrupts can be delivered - * with IR=1/DR=1, if AIL==2 and MSR.HV won't be changed by delivering - * it. Addresses are the same as the original interrupt addresses, but - * offset by 0xc000000000004000. - * It's impossible to receive interrupts below 0x300 via this mechanism. - * KVM: None of these traps are from the guest ; anything that escalated - * to HV=1 from HV=0 is delivered via real mode handlers. - */ - - /* - * This uses the standard macro, since the original 0x300 vector - * only has extra guff for STAB-based processors -- which never - * come here. - */ - STD_RELON_EXCEPTION_PSERIES(0x4300, 0x300, data_access) - . = 0x4380 - .globl data_access_slb_relon_pSeries -data_access_slb_relon_pSeries: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380) - std r3,PACA_EXSLB+EX_R3(r13) - mfspr r3,SPRN_DAR - mfspr r12,SPRN_SRR1 -#ifndef CONFIG_RELOCATABLE - b slb_miss_realmode -#else - /* - * We can't just use a direct branch to slb_miss_realmode - * because the distance from here to there depends on where - * the kernel ends up being put. - */ - mfctr r11 - ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, slb_miss_realmode) - mtctr r10 - bctr -#endif - - STD_RELON_EXCEPTION_PSERIES(0x4400, 0x400, instruction_access) - . = 0x4480 - .globl instruction_access_slb_relon_pSeries -instruction_access_slb_relon_pSeries: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x480) - std r3,PACA_EXSLB+EX_R3(r13) - mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ - mfspr r12,SPRN_SRR1 -#ifndef CONFIG_RELOCATABLE - b slb_miss_realmode -#else - mfctr r11 - ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, slb_miss_realmode) - mtctr r10 - bctr -#endif - - . = 0x4500 - .globl hardware_interrupt_relon_pSeries; - .globl hardware_interrupt_relon_hv; -hardware_interrupt_relon_pSeries: -hardware_interrupt_relon_hv: - BEGIN_FTR_SECTION - _MASKABLE_RELON_EXCEPTION_PSERIES(0x502, hardware_interrupt, EXC_HV, SOFTEN_TEST_HV) - FTR_SECTION_ELSE - _MASKABLE_RELON_EXCEPTION_PSERIES(0x500, hardware_interrupt, EXC_STD, SOFTEN_TEST_PR) - ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE) - STD_RELON_EXCEPTION_PSERIES(0x4600, 0x600, alignment) - STD_RELON_EXCEPTION_PSERIES(0x4700, 0x700, program_check) - STD_RELON_EXCEPTION_PSERIES(0x4800, 0x800, fp_unavailable) - MASKABLE_RELON_EXCEPTION_PSERIES(0x4900, 0x900, decrementer) - STD_RELON_EXCEPTION_HV(0x4980, 0x982, hdecrementer) - MASKABLE_RELON_EXCEPTION_PSERIES(0x4a00, 0xa00, doorbell_super) - STD_RELON_EXCEPTION_PSERIES(0x4b00, 0xb00, trap_0b) - - . = 0x4c00 - .globl system_call_relon_pSeries -system_call_relon_pSeries: - HMT_MEDIUM - SYSCALL_PSERIES_1 - SYSCALL_PSERIES_2_DIRECT - SYSCALL_PSERIES_3 - - STD_RELON_EXCEPTION_PSERIES(0x4d00, 0xd00, single_step) - - . = 0x4e00 - b . /* Can't happen, see v2.07 Book III-S section 6.5 */ - - . = 0x4e20 - b . /* Can't happen, see v2.07 Book III-S section 6.5 */ - - . = 0x4e40 -emulation_assist_relon_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b emulation_assist_relon_hv - - . = 0x4e60 - b . /* Can't happen, see v2.07 Book III-S section 6.5 */ - - . = 0x4e80 -h_doorbell_relon_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_doorbell_relon_hv - - . = 0x4ea0 -h_virt_irq_relon_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b h_virt_irq_relon_hv - - . = 0x4f00 -performance_monitor_relon_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b performance_monitor_relon_pSeries - - . = 0x4f20 -altivec_unavailable_relon_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b altivec_unavailable_relon_pSeries - - . = 0x4f40 -vsx_unavailable_relon_pseries_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b vsx_unavailable_relon_pSeries - - . = 0x4f60 -facility_unavailable_relon_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b facility_unavailable_relon_pSeries - - . = 0x4f80 -hv_facility_unavailable_relon_trampoline: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b hv_facility_unavailable_relon_hv - - STD_RELON_EXCEPTION_PSERIES(0x5300, 0x1300, instruction_breakpoint) -#ifdef CONFIG_PPC_DENORMALISATION - . = 0x5500 - b denorm_exception_hv -#endif - STD_RELON_EXCEPTION_PSERIES(0x5700, 0x1700, altivec_assist) - - .align 7 -system_call_entry: - b system_call_common - -ppc64_runlatch_on_trampoline: - b __ppc64_runlatch_on - -/* - * Here r13 points to the paca, r9 contains the saved CR, - * SRR0 and SRR1 are saved in r11 and r12, - * r9 - r13 are saved in paca->exgen. - */ - .align 7 - .globl data_access_common -data_access_common: - mfspr r10,SPRN_DAR - std r10,PACA_EXGEN+EX_DAR(r13) - mfspr r10,SPRN_DSISR - stw r10,PACA_EXGEN+EX_DSISR(r13) - EXCEPTION_PROLOG_COMMON(0x300, PACA_EXGEN) - RECONCILE_IRQ_STATE(r10, r11) - ld r12,_MSR(r1) - ld r3,PACA_EXGEN+EX_DAR(r13) - lwz r4,PACA_EXGEN+EX_DSISR(r13) - li r5,0x300 - std r3,_DAR(r1) - std r4,_DSISR(r1) -BEGIN_MMU_FTR_SECTION - b do_hash_page /* Try to handle as hpte fault */ -MMU_FTR_SECTION_ELSE - b handle_page_fault -ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) - - .align 7 - .globl h_data_storage_common -h_data_storage_common: - mfspr r10,SPRN_HDAR - std r10,PACA_EXGEN+EX_DAR(r13) - mfspr r10,SPRN_HDSISR - stw r10,PACA_EXGEN+EX_DSISR(r13) - EXCEPTION_PROLOG_COMMON(0xe00, PACA_EXGEN) - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl unknown_exception - b ret_from_except - - .align 7 - .globl instruction_access_common -instruction_access_common: - EXCEPTION_PROLOG_COMMON(0x400, PACA_EXGEN) - RECONCILE_IRQ_STATE(r10, r11) - ld r12,_MSR(r1) - ld r3,_NIP(r1) - andis. r4,r12,0x5820 - li r5,0x400 - std r3,_DAR(r1) - std r4,_DSISR(r1) -BEGIN_MMU_FTR_SECTION - b do_hash_page /* Try to handle as hpte fault */ -MMU_FTR_SECTION_ELSE - b handle_page_fault -ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) - - STD_EXCEPTION_COMMON(0xe20, h_instr_storage, unknown_exception) +TRAMP_KVM_SKIP(PACA_EXMC, 0x200) +EXC_COMMON_BEGIN(machine_check_common) /* * Machine check is different because we use a different * save area: PACA_EXMC instead of PACA_EXGEN. */ - .align 7 - .globl machine_check_common -machine_check_common: - mfspr r10,SPRN_DAR std r10,PACA_EXMC+EX_DAR(r13) mfspr r10,SPRN_DSISR @@ -1003,230 +294,6 @@ machine_check_common: bl machine_check_exception b ret_from_except - .align 7 - .globl alignment_common -alignment_common: - mfspr r10,SPRN_DAR - std r10,PACA_EXGEN+EX_DAR(r13) - mfspr r10,SPRN_DSISR - stw r10,PACA_EXGEN+EX_DSISR(r13) - EXCEPTION_PROLOG_COMMON(0x600, PACA_EXGEN) - ld r3,PACA_EXGEN+EX_DAR(r13) - lwz r4,PACA_EXGEN+EX_DSISR(r13) - std r3,_DAR(r1) - std r4,_DSISR(r1) - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl alignment_exception - b ret_from_except - - .align 7 - .globl program_check_common -program_check_common: - EXCEPTION_PROLOG_COMMON(0x700, PACA_EXGEN) - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl program_check_exception - b ret_from_except - - .align 7 - .globl fp_unavailable_common -fp_unavailable_common: - EXCEPTION_PROLOG_COMMON(0x800, PACA_EXGEN) - bne 1f /* if from user, just load it up */ - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl kernel_fp_unavailable_exception - BUG_OPCODE -1: -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -BEGIN_FTR_SECTION - /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in - * transaction), go do TM stuff - */ - rldicl. r0, r12, (64-MSR_TS_LG), (64-2) - bne- 2f -END_FTR_SECTION_IFSET(CPU_FTR_TM) -#endif - bl load_up_fpu - b fast_exception_return -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -2: /* User process was in a transaction */ - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl fp_unavailable_tm - b ret_from_except -#endif - .align 7 - .globl altivec_unavailable_common -altivec_unavailable_common: - EXCEPTION_PROLOG_COMMON(0xf20, PACA_EXGEN) -#ifdef CONFIG_ALTIVEC -BEGIN_FTR_SECTION - beq 1f -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - BEGIN_FTR_SECTION_NESTED(69) - /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in - * transaction), go do TM stuff - */ - rldicl. r0, r12, (64-MSR_TS_LG), (64-2) - bne- 2f - END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) -#endif - bl load_up_altivec - b fast_exception_return -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -2: /* User process was in a transaction */ - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl altivec_unavailable_tm - b ret_from_except -#endif -1: -END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) -#endif - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl altivec_unavailable_exception - b ret_from_except - - .align 7 - .globl vsx_unavailable_common -vsx_unavailable_common: - EXCEPTION_PROLOG_COMMON(0xf40, PACA_EXGEN) -#ifdef CONFIG_VSX -BEGIN_FTR_SECTION - beq 1f -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - BEGIN_FTR_SECTION_NESTED(69) - /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in - * transaction), go do TM stuff - */ - rldicl. r0, r12, (64-MSR_TS_LG), (64-2) - bne- 2f - END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) -#endif - b load_up_vsx -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -2: /* User process was in a transaction */ - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl vsx_unavailable_tm - b ret_from_except -#endif -1: -END_FTR_SECTION_IFSET(CPU_FTR_VSX) -#endif - bl save_nvgprs - RECONCILE_IRQ_STATE(r10, r11) - addi r3,r1,STACK_FRAME_OVERHEAD - bl vsx_unavailable_exception - b ret_from_except - - /* Equivalents to the above handlers for relocation-on interrupt vectors */ - STD_RELON_EXCEPTION_HV_OOL(0xe40, emulation_assist) - MASKABLE_RELON_EXCEPTION_HV_OOL(0xe80, h_doorbell) - MASKABLE_RELON_EXCEPTION_HV_OOL(0xea0, h_virt_irq) - - STD_RELON_EXCEPTION_PSERIES_OOL(0xf00, performance_monitor) - STD_RELON_EXCEPTION_PSERIES_OOL(0xf20, altivec_unavailable) - STD_RELON_EXCEPTION_PSERIES_OOL(0xf40, vsx_unavailable) - STD_RELON_EXCEPTION_PSERIES_OOL(0xf60, facility_unavailable) - STD_RELON_EXCEPTION_HV_OOL(0xf80, hv_facility_unavailable) - - /* - * The __end_interrupts marker must be past the out-of-line (OOL) - * handlers, so that they are copied to real address 0x100 when running - * a relocatable kernel. This ensures they can be reached from the short - * trampoline handlers (like 0x4f00, 0x4f20, etc.) which branch - * directly, without using LOAD_HANDLER(). - */ - .align 7 - .globl __end_interrupts -__end_interrupts: - -#if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) -/* - * Data area reserved for FWNMI option. - * This address (0x7000) is fixed by the RPA. - */ - .= 0x7000 - .globl fwnmi_data_area -fwnmi_data_area: - - /* pseries and powernv need to keep the whole page from - * 0x7000 to 0x8000 free for use by the firmware - */ - . = 0x8000 -#endif /* defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) */ - - STD_EXCEPTION_COMMON(0xf60, facility_unavailable, facility_unavailable_exception) - STD_EXCEPTION_COMMON(0xf80, hv_facility_unavailable, facility_unavailable_exception) - -#ifdef CONFIG_CBE_RAS - STD_EXCEPTION_COMMON(0x1200, cbe_system_error, cbe_system_error_exception) - STD_EXCEPTION_COMMON(0x1600, cbe_maintenance, cbe_maintenance_exception) - STD_EXCEPTION_COMMON(0x1800, cbe_thermal, cbe_thermal_exception) -#endif /* CONFIG_CBE_RAS */ - - .globl hmi_exception_early -hmi_exception_early: - EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST, 0xe62) - mr r10,r1 /* Save r1 */ - ld r1,PACAEMERGSP(r13) /* Use emergency stack */ - subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */ - std r9,_CCR(r1) /* save CR in stackframe */ - mfspr r11,SPRN_HSRR0 /* Save HSRR0 */ - std r11,_NIP(r1) /* save HSRR0 in stackframe */ - mfspr r12,SPRN_HSRR1 /* Save SRR1 */ - std r12,_MSR(r1) /* save SRR1 in stackframe */ - std r10,0(r1) /* make stack chain pointer */ - std r0,GPR0(r1) /* save r0 in stackframe */ - std r10,GPR1(r1) /* save r1 in stackframe */ - EXCEPTION_PROLOG_COMMON_2(PACA_EXGEN) - EXCEPTION_PROLOG_COMMON_3(0xe60) - addi r3,r1,STACK_FRAME_OVERHEAD - bl hmi_exception_realmode - /* Windup the stack. */ - /* Move original HSRR0 and HSRR1 into the respective regs */ - ld r9,_MSR(r1) - mtspr SPRN_HSRR1,r9 - ld r3,_NIP(r1) - mtspr SPRN_HSRR0,r3 - ld r9,_CTR(r1) - mtctr r9 - ld r9,_XER(r1) - mtxer r9 - ld r9,_LINK(r1) - mtlr r9 - REST_GPR(0, r1) - REST_8GPRS(2, r1) - REST_GPR(10, r1) - ld r11,_CCR(r1) - mtcr r11 - REST_GPR(11, r1) - REST_2GPRS(12, r1) - /* restore original r1. */ - ld r1,GPR1(r1) - - /* - * Go to virtual mode and pull the HMI event information from - * firmware. - */ - .globl hmi_exception_after_realmode -hmi_exception_after_realmode: - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXGEN) - b hmi_exception_hv - - #define MACHINE_CHECK_HANDLER_WINDUP \ /* Clear MSR_RI before setting SRR0 and SRR1. */\ li r0,MSR_RI; \ @@ -1262,9 +329,7 @@ hmi_exception_after_realmode: * Handle machine check early in real mode. We come here with * ME=1, MMU (IR=0 and DR=0) off and using MC emergency stack. */ - .align 7 - .globl machine_check_handle_early -machine_check_handle_early: +EXC_COMMON_BEGIN(machine_check_handle_early) std r0,GPR0(r1) /* Save r0 */ EXCEPTION_PROLOG_COMMON_3(0x200) bl save_nvgprs @@ -1353,7 +418,6 @@ machine_check_handle_early: andi. r11,r12,MSR_RI bne 2f 1: mfspr r11,SPRN_SRR0 - ld r10,PACAKBASE(r13) LOAD_HANDLER(r10,unrecover_mce) mtspr SPRN_SRR0,r10 ld r10,PACAKMSR(r13) @@ -1391,7 +455,7 @@ machine_check_handle_early: MACHINE_CHECK_HANDLER_WINDUP b machine_check_pSeries -unrecover_mce: +EXC_COMMON_BEGIN(unrecover_mce) /* Invoke machine_check_exception to print MCE event and panic. */ addi r3,r1,STACK_FRAME_OVERHEAD bl machine_check_exception @@ -1402,15 +466,155 @@ unrecover_mce: 1: addi r3,r1,STACK_FRAME_OVERHEAD bl unrecoverable_exception b 1b -/* - * r13 points to the PACA, r9 contains the saved CR, - * r12 contain the saved SRR1, SRR0 is still ready for return - * r3 has the faulting address - * r9 - r13 are saved in paca->exslb. - * r3 is saved in paca->slb_r3 - * We assume we aren't going to take any exceptions during this procedure. - */ -slb_miss_realmode: + + +EXC_REAL(data_access, 0x300, 0x380) +EXC_VIRT(data_access, 0x4300, 0x4380, 0x300) +TRAMP_KVM_SKIP(PACA_EXGEN, 0x300) + +EXC_COMMON_BEGIN(data_access_common) + /* + * Here r13 points to the paca, r9 contains the saved CR, + * SRR0 and SRR1 are saved in r11 and r12, + * r9 - r13 are saved in paca->exgen. + */ + mfspr r10,SPRN_DAR + std r10,PACA_EXGEN+EX_DAR(r13) + mfspr r10,SPRN_DSISR + stw r10,PACA_EXGEN+EX_DSISR(r13) + EXCEPTION_PROLOG_COMMON(0x300, PACA_EXGEN) + RECONCILE_IRQ_STATE(r10, r11) + ld r12,_MSR(r1) + ld r3,PACA_EXGEN+EX_DAR(r13) + lwz r4,PACA_EXGEN+EX_DSISR(r13) + li r5,0x300 + std r3,_DAR(r1) + std r4,_DSISR(r1) +BEGIN_MMU_FTR_SECTION + b do_hash_page /* Try to handle as hpte fault */ +MMU_FTR_SECTION_ELSE + b handle_page_fault +ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) + + +EXC_REAL_BEGIN(data_access_slb, 0x380, 0x400) + SET_SCRATCH0(r13) + EXCEPTION_PROLOG_0(PACA_EXSLB) + EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x380) + std r3,PACA_EXSLB+EX_R3(r13) + mfspr r3,SPRN_DAR + mfspr r12,SPRN_SRR1 + crset 4*cr6+eq +#ifndef CONFIG_RELOCATABLE + b slb_miss_realmode +#else + /* + * We can't just use a direct branch to slb_miss_realmode + * because the distance from here to there depends on where + * the kernel ends up being put. + */ + mfctr r11 + LOAD_HANDLER(r10, slb_miss_realmode) + mtctr r10 + bctr +#endif +EXC_REAL_END(data_access_slb, 0x380, 0x400) + +EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x4400) + SET_SCRATCH0(r13) + EXCEPTION_PROLOG_0(PACA_EXSLB) + EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380) + std r3,PACA_EXSLB+EX_R3(r13) + mfspr r3,SPRN_DAR + mfspr r12,SPRN_SRR1 + crset 4*cr6+eq +#ifndef CONFIG_RELOCATABLE + b slb_miss_realmode +#else + /* + * We can't just use a direct branch to slb_miss_realmode + * because the distance from here to there depends on where + * the kernel ends up being put. + */ + mfctr r11 + LOAD_HANDLER(r10, slb_miss_realmode) + mtctr r10 + bctr +#endif +EXC_VIRT_END(data_access_slb, 0x4380, 0x4400) +TRAMP_KVM_SKIP(PACA_EXSLB, 0x380) + + +EXC_REAL(instruction_access, 0x400, 0x480) +EXC_VIRT(instruction_access, 0x4400, 0x4480, 0x400) +TRAMP_KVM(PACA_EXGEN, 0x400) + +EXC_COMMON_BEGIN(instruction_access_common) + EXCEPTION_PROLOG_COMMON(0x400, PACA_EXGEN) + RECONCILE_IRQ_STATE(r10, r11) + ld r12,_MSR(r1) + ld r3,_NIP(r1) + andis. r4,r12,0x5820 + li r5,0x400 + std r3,_DAR(r1) + std r4,_DSISR(r1) +BEGIN_MMU_FTR_SECTION + b do_hash_page /* Try to handle as hpte fault */ +MMU_FTR_SECTION_ELSE + b handle_page_fault +ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) + + +EXC_REAL_BEGIN(instruction_access_slb, 0x480, 0x500) + SET_SCRATCH0(r13) + EXCEPTION_PROLOG_0(PACA_EXSLB) + EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x480) + std r3,PACA_EXSLB+EX_R3(r13) + mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ + mfspr r12,SPRN_SRR1 + crclr 4*cr6+eq +#ifndef CONFIG_RELOCATABLE + b slb_miss_realmode +#else + mfctr r11 + LOAD_HANDLER(r10, slb_miss_realmode) + mtctr r10 + bctr +#endif +EXC_REAL_END(instruction_access_slb, 0x480, 0x500) + +EXC_VIRT_BEGIN(instruction_access_slb, 0x4480, 0x4500) + SET_SCRATCH0(r13) + EXCEPTION_PROLOG_0(PACA_EXSLB) + EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x480) + std r3,PACA_EXSLB+EX_R3(r13) + mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ + mfspr r12,SPRN_SRR1 + crclr 4*cr6+eq +#ifndef CONFIG_RELOCATABLE + b slb_miss_realmode +#else + mfctr r11 + LOAD_HANDLER(r10, slb_miss_realmode) + mtctr r10 + bctr +#endif +EXC_VIRT_END(instruction_access_slb, 0x4480, 0x4500) +TRAMP_KVM(PACA_EXSLB, 0x480) + + +/* This handler is used by both 0x380 and 0x480 slb miss interrupts */ +EXC_COMMON_BEGIN(slb_miss_realmode) + /* + * r13 points to the PACA, r9 contains the saved CR, + * r12 contain the saved SRR1, SRR0 is still ready for return + * r3 has the faulting address + * r9 - r13 are saved in paca->exslb. + * r3 is saved in paca->slb_r3 + * cr6.eq is set for a D-SLB miss, clear for a I-SLB miss + * We assume we aren't going to take any exceptions during this + * procedure. + */ mflr r10 #ifdef CONFIG_RELOCATABLE mtctr r11 @@ -1418,29 +622,31 @@ slb_miss_realmode: stw r9,PACA_EXSLB+EX_CCR(r13) /* save CR in exc. frame */ std r10,PACA_EXSLB+EX_LR(r13) /* save LR */ + std r3,PACA_EXSLB+EX_DAR(r13) + crset 4*cr0+eq #ifdef CONFIG_PPC_STD_MMU_64 BEGIN_MMU_FTR_SECTION bl slb_allocate_realmode END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX) #endif - /* All done -- return from exception. */ ld r10,PACA_EXSLB+EX_LR(r13) ld r3,PACA_EXSLB+EX_R3(r13) lwz r9,PACA_EXSLB+EX_CCR(r13) /* get saved CR */ - mtlr r10 + + beq 8f /* if bad address, make full stack frame */ + andi. r10,r12,MSR_RI /* check for unrecoverable exception */ -BEGIN_MMU_FTR_SECTION beq- 2f -FTR_SECTION_ELSE - b 2f -ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) + + /* All done -- return from exception. */ .machine push .machine "power4" mtcrf 0x80,r9 + mtcrf 0x02,r9 /* I/D indication is in cr6 */ mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */ .machine pop @@ -1454,7 +660,6 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) b . /* prevent speculative execution */ 2: mfspr r11,SPRN_SRR0 - ld r10,PACAKBASE(r13) LOAD_HANDLER(r10,unrecov_slb) mtspr SPRN_SRR0,r10 ld r10,PACAKMSR(r13) @@ -1462,7 +667,15 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) rfid b . -unrecov_slb: +8: mfspr r11,SPRN_SRR0 + LOAD_HANDLER(r10,bad_addr_slb) + mtspr SPRN_SRR0,r10 + ld r10,PACAKMSR(r13) + mtspr SPRN_SRR1,r10 + rfid + b . + +EXC_COMMON_BEGIN(unrecov_slb) EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB) RECONCILE_IRQ_STATE(r10, r11) bl save_nvgprs @@ -1470,9 +683,701 @@ unrecov_slb: bl unrecoverable_exception b 1b +EXC_COMMON_BEGIN(bad_addr_slb) + EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB) + RECONCILE_IRQ_STATE(r10, r11) + ld r3, PACA_EXSLB+EX_DAR(r13) + std r3, _DAR(r1) + beq cr6, 2f + li r10, 0x480 /* fix trap number for I-SLB miss */ + std r10, _TRAP(r1) +2: bl save_nvgprs + addi r3, r1, STACK_FRAME_OVERHEAD + bl slb_miss_bad_addr + b ret_from_except + +EXC_REAL_BEGIN(hardware_interrupt, 0x500, 0x600) + .globl hardware_interrupt_hv; +hardware_interrupt_hv: + BEGIN_FTR_SECTION + _MASKABLE_EXCEPTION_PSERIES(0x500, hardware_interrupt_common, + EXC_HV, SOFTEN_TEST_HV) +do_kvm_H0x500: + KVM_HANDLER(PACA_EXGEN, EXC_HV, 0x502) + FTR_SECTION_ELSE + _MASKABLE_EXCEPTION_PSERIES(0x500, hardware_interrupt_common, + EXC_STD, SOFTEN_TEST_PR) +do_kvm_0x500: + KVM_HANDLER(PACA_EXGEN, EXC_STD, 0x500) + ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) +EXC_REAL_END(hardware_interrupt, 0x500, 0x600) + +EXC_VIRT_BEGIN(hardware_interrupt, 0x4500, 0x4600) + .globl hardware_interrupt_relon_hv; +hardware_interrupt_relon_hv: + BEGIN_FTR_SECTION + _MASKABLE_RELON_EXCEPTION_PSERIES(0x500, hardware_interrupt_common, EXC_HV, SOFTEN_TEST_HV) + FTR_SECTION_ELSE + _MASKABLE_RELON_EXCEPTION_PSERIES(0x500, hardware_interrupt_common, EXC_STD, SOFTEN_TEST_PR) + ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE) +EXC_VIRT_END(hardware_interrupt, 0x4500, 0x4600) + +EXC_COMMON_ASYNC(hardware_interrupt_common, 0x500, do_IRQ) + + +EXC_REAL(alignment, 0x600, 0x700) +EXC_VIRT(alignment, 0x4600, 0x4700, 0x600) +TRAMP_KVM(PACA_EXGEN, 0x600) +EXC_COMMON_BEGIN(alignment_common) + mfspr r10,SPRN_DAR + std r10,PACA_EXGEN+EX_DAR(r13) + mfspr r10,SPRN_DSISR + stw r10,PACA_EXGEN+EX_DSISR(r13) + EXCEPTION_PROLOG_COMMON(0x600, PACA_EXGEN) + ld r3,PACA_EXGEN+EX_DAR(r13) + lwz r4,PACA_EXGEN+EX_DSISR(r13) + std r3,_DAR(r1) + std r4,_DSISR(r1) + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl alignment_exception + b ret_from_except + + +EXC_REAL(program_check, 0x700, 0x800) +EXC_VIRT(program_check, 0x4700, 0x4800, 0x700) +TRAMP_KVM(PACA_EXGEN, 0x700) +EXC_COMMON_BEGIN(program_check_common) + EXCEPTION_PROLOG_COMMON(0x700, PACA_EXGEN) + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl program_check_exception + b ret_from_except + + +EXC_REAL(fp_unavailable, 0x800, 0x900) +EXC_VIRT(fp_unavailable, 0x4800, 0x4900, 0x800) +TRAMP_KVM(PACA_EXGEN, 0x800) +EXC_COMMON_BEGIN(fp_unavailable_common) + EXCEPTION_PROLOG_COMMON(0x800, PACA_EXGEN) + bne 1f /* if from user, just load it up */ + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl kernel_fp_unavailable_exception + BUG_OPCODE +1: +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +BEGIN_FTR_SECTION + /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in + * transaction), go do TM stuff + */ + rldicl. r0, r12, (64-MSR_TS_LG), (64-2) + bne- 2f +END_FTR_SECTION_IFSET(CPU_FTR_TM) +#endif + bl load_up_fpu + b fast_exception_return +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +2: /* User process was in a transaction */ + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl fp_unavailable_tm + b ret_from_except +#endif + + +EXC_REAL_MASKABLE(decrementer, 0x900, 0x980) +EXC_VIRT_MASKABLE(decrementer, 0x4900, 0x4980, 0x900) +TRAMP_KVM(PACA_EXGEN, 0x900) +EXC_COMMON_ASYNC(decrementer_common, 0x900, timer_interrupt) + + +EXC_REAL_HV(hdecrementer, 0x980, 0xa00) +EXC_VIRT_HV(hdecrementer, 0x4980, 0x4a00, 0x980) +TRAMP_KVM_HV(PACA_EXGEN, 0x980) +EXC_COMMON(hdecrementer_common, 0x980, hdec_interrupt) + + +EXC_REAL_MASKABLE(doorbell_super, 0xa00, 0xb00) +EXC_VIRT_MASKABLE(doorbell_super, 0x4a00, 0x4b00, 0xa00) +TRAMP_KVM(PACA_EXGEN, 0xa00) +#ifdef CONFIG_PPC_DOORBELL +EXC_COMMON_ASYNC(doorbell_super_common, 0xa00, doorbell_exception) +#else +EXC_COMMON_ASYNC(doorbell_super_common, 0xa00, unknown_exception) +#endif + + +EXC_REAL(trap_0b, 0xb00, 0xc00) +EXC_VIRT(trap_0b, 0x4b00, 0x4c00, 0xb00) +TRAMP_KVM(PACA_EXGEN, 0xb00) +EXC_COMMON(trap_0b_common, 0xb00, unknown_exception) + + +#define LOAD_SYSCALL_HANDLER(reg) \ + ld reg,PACAKBASE(r13); \ + ori reg,reg,(ABS_ADDR(system_call_common))@l; + +/* Syscall routine is used twice, in reloc-off and reloc-on paths */ +#define SYSCALL_PSERIES_1 \ +BEGIN_FTR_SECTION \ + cmpdi r0,0x1ebe ; \ + beq- 1f ; \ +END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) \ + mr r9,r13 ; \ + GET_PACA(r13) ; \ + mfspr r11,SPRN_SRR0 ; \ +0: + +#define SYSCALL_PSERIES_2_RFID \ + mfspr r12,SPRN_SRR1 ; \ + LOAD_SYSCALL_HANDLER(r10) ; \ + mtspr SPRN_SRR0,r10 ; \ + ld r10,PACAKMSR(r13) ; \ + mtspr SPRN_SRR1,r10 ; \ + rfid ; \ + b . ; /* prevent speculative execution */ + +#define SYSCALL_PSERIES_3 \ + /* Fast LE/BE switch system call */ \ +1: mfspr r12,SPRN_SRR1 ; \ + xori r12,r12,MSR_LE ; \ + mtspr SPRN_SRR1,r12 ; \ + rfid ; /* return to userspace */ \ + b . ; /* prevent speculative execution */ + +#if defined(CONFIG_RELOCATABLE) + /* + * We can't branch directly so we do it via the CTR which + * is volatile across system calls. + */ +#define SYSCALL_PSERIES_2_DIRECT \ + LOAD_SYSCALL_HANDLER(r12) ; \ + mtctr r12 ; \ + mfspr r12,SPRN_SRR1 ; \ + li r10,MSR_RI ; \ + mtmsrd r10,1 ; \ + bctr ; +#else + /* We can branch directly */ +#define SYSCALL_PSERIES_2_DIRECT \ + mfspr r12,SPRN_SRR1 ; \ + li r10,MSR_RI ; \ + mtmsrd r10,1 ; /* Set RI (EE=0) */ \ + b system_call_common ; +#endif + +EXC_REAL_BEGIN(system_call, 0xc00, 0xd00) + /* + * If CONFIG_KVM_BOOK3S_64_HANDLER is set, save the PPR (on systems + * that support it) before changing to HMT_MEDIUM. That allows the KVM + * code to save that value into the guest state (it is the guest's PPR + * value). Otherwise just change to HMT_MEDIUM as userspace has + * already saved the PPR. + */ +#ifdef CONFIG_KVM_BOOK3S_64_HANDLER + SET_SCRATCH0(r13) + GET_PACA(r13) + std r9,PACA_EXGEN+EX_R9(r13) + OPT_GET_SPR(r9, SPRN_PPR, CPU_FTR_HAS_PPR); + HMT_MEDIUM; + std r10,PACA_EXGEN+EX_R10(r13) + OPT_SAVE_REG_TO_PACA(PACA_EXGEN+EX_PPR, r9, CPU_FTR_HAS_PPR); + mfcr r9 + KVMTEST_PR(0xc00) + GET_SCRATCH0(r13) +#else + HMT_MEDIUM; +#endif + SYSCALL_PSERIES_1 + SYSCALL_PSERIES_2_RFID + SYSCALL_PSERIES_3 +EXC_REAL_END(system_call, 0xc00, 0xd00) + +EXC_VIRT_BEGIN(system_call, 0x4c00, 0x4d00) + HMT_MEDIUM + SYSCALL_PSERIES_1 + SYSCALL_PSERIES_2_DIRECT + SYSCALL_PSERIES_3 +EXC_VIRT_END(system_call, 0x4c00, 0x4d00) + +TRAMP_KVM(PACA_EXGEN, 0xc00) + + +EXC_REAL(single_step, 0xd00, 0xe00) +EXC_VIRT(single_step, 0x4d00, 0x4e00, 0xd00) +TRAMP_KVM(PACA_EXGEN, 0xd00) +EXC_COMMON(single_step_common, 0xd00, single_step_exception) + +EXC_REAL_OOL_HV(h_data_storage, 0xe00, 0xe20) +EXC_VIRT_NONE(0x4e00, 0x4e20) +TRAMP_KVM_HV_SKIP(PACA_EXGEN, 0xe00) +EXC_COMMON_BEGIN(h_data_storage_common) + mfspr r10,SPRN_HDAR + std r10,PACA_EXGEN+EX_DAR(r13) + mfspr r10,SPRN_HDSISR + stw r10,PACA_EXGEN+EX_DSISR(r13) + EXCEPTION_PROLOG_COMMON(0xe00, PACA_EXGEN) + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl unknown_exception + b ret_from_except + + +EXC_REAL_OOL_HV(h_instr_storage, 0xe20, 0xe40) +EXC_VIRT_NONE(0x4e20, 0x4e40) +TRAMP_KVM_HV(PACA_EXGEN, 0xe20) +EXC_COMMON(h_instr_storage_common, 0xe20, unknown_exception) + + +EXC_REAL_OOL_HV(emulation_assist, 0xe40, 0xe60) +EXC_VIRT_OOL_HV(emulation_assist, 0x4e40, 0x4e60, 0xe40) +TRAMP_KVM_HV(PACA_EXGEN, 0xe40) +EXC_COMMON(emulation_assist_common, 0xe40, emulation_assist_interrupt) + + +/* + * hmi_exception trampoline is a special case. It jumps to hmi_exception_early + * first, and then eventaully from there to the trampoline to get into virtual + * mode. + */ +__EXC_REAL_OOL_HV_DIRECT(hmi_exception, 0xe60, 0xe80, hmi_exception_early) +__TRAMP_REAL_REAL_OOL_MASKABLE_HV(hmi_exception, 0xe60) +EXC_VIRT_NONE(0x4e60, 0x4e80) +TRAMP_KVM_HV(PACA_EXGEN, 0xe60) +TRAMP_REAL_BEGIN(hmi_exception_early) + EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_HV, 0xe60) + mr r10,r1 /* Save r1 */ + ld r1,PACAEMERGSP(r13) /* Use emergency stack */ + subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */ + std r9,_CCR(r1) /* save CR in stackframe */ + mfspr r11,SPRN_HSRR0 /* Save HSRR0 */ + std r11,_NIP(r1) /* save HSRR0 in stackframe */ + mfspr r12,SPRN_HSRR1 /* Save SRR1 */ + std r12,_MSR(r1) /* save SRR1 in stackframe */ + std r10,0(r1) /* make stack chain pointer */ + std r0,GPR0(r1) /* save r0 in stackframe */ + std r10,GPR1(r1) /* save r1 in stackframe */ + EXCEPTION_PROLOG_COMMON_2(PACA_EXGEN) + EXCEPTION_PROLOG_COMMON_3(0xe60) + addi r3,r1,STACK_FRAME_OVERHEAD + bl hmi_exception_realmode + /* Windup the stack. */ + /* Move original HSRR0 and HSRR1 into the respective regs */ + ld r9,_MSR(r1) + mtspr SPRN_HSRR1,r9 + ld r3,_NIP(r1) + mtspr SPRN_HSRR0,r3 + ld r9,_CTR(r1) + mtctr r9 + ld r9,_XER(r1) + mtxer r9 + ld r9,_LINK(r1) + mtlr r9 + REST_GPR(0, r1) + REST_8GPRS(2, r1) + REST_GPR(10, r1) + ld r11,_CCR(r1) + mtcr r11 + REST_GPR(11, r1) + REST_2GPRS(12, r1) + /* restore original r1. */ + ld r1,GPR1(r1) + + /* + * Go to virtual mode and pull the HMI event information from + * firmware. + */ + .globl hmi_exception_after_realmode +hmi_exception_after_realmode: + SET_SCRATCH0(r13) + EXCEPTION_PROLOG_0(PACA_EXGEN) + b tramp_real_hmi_exception + +EXC_COMMON_ASYNC(hmi_exception_common, 0xe60, handle_hmi_exception) + + +EXC_REAL_OOL_MASKABLE_HV(h_doorbell, 0xe80, 0xea0) +EXC_VIRT_OOL_MASKABLE_HV(h_doorbell, 0x4e80, 0x4ea0, 0xe80) +TRAMP_KVM_HV(PACA_EXGEN, 0xe80) +#ifdef CONFIG_PPC_DOORBELL +EXC_COMMON_ASYNC(h_doorbell_common, 0xe80, doorbell_exception) +#else +EXC_COMMON_ASYNC(h_doorbell_common, 0xe80, unknown_exception) +#endif + + +EXC_REAL_OOL_MASKABLE_HV(h_virt_irq, 0xea0, 0xec0) +EXC_VIRT_OOL_MASKABLE_HV(h_virt_irq, 0x4ea0, 0x4ec0, 0xea0) +TRAMP_KVM_HV(PACA_EXGEN, 0xea0) +EXC_COMMON_ASYNC(h_virt_irq_common, 0xea0, do_IRQ) + + +EXC_REAL_NONE(0xec0, 0xf00) +EXC_VIRT_NONE(0x4ec0, 0x4f00) + + +EXC_REAL_OOL(performance_monitor, 0xf00, 0xf20) +EXC_VIRT_OOL(performance_monitor, 0x4f00, 0x4f20, 0xf00) +TRAMP_KVM(PACA_EXGEN, 0xf00) +EXC_COMMON_ASYNC(performance_monitor_common, 0xf00, performance_monitor_exception) + + +EXC_REAL_OOL(altivec_unavailable, 0xf20, 0xf40) +EXC_VIRT_OOL(altivec_unavailable, 0x4f20, 0x4f40, 0xf20) +TRAMP_KVM(PACA_EXGEN, 0xf20) +EXC_COMMON_BEGIN(altivec_unavailable_common) + EXCEPTION_PROLOG_COMMON(0xf20, PACA_EXGEN) +#ifdef CONFIG_ALTIVEC +BEGIN_FTR_SECTION + beq 1f +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + BEGIN_FTR_SECTION_NESTED(69) + /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in + * transaction), go do TM stuff + */ + rldicl. r0, r12, (64-MSR_TS_LG), (64-2) + bne- 2f + END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) +#endif + bl load_up_altivec + b fast_exception_return +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +2: /* User process was in a transaction */ + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl altivec_unavailable_tm + b ret_from_except +#endif +1: +END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) +#endif + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl altivec_unavailable_exception + b ret_from_except + + +EXC_REAL_OOL(vsx_unavailable, 0xf40, 0xf60) +EXC_VIRT_OOL(vsx_unavailable, 0x4f40, 0x4f60, 0xf40) +TRAMP_KVM(PACA_EXGEN, 0xf40) +EXC_COMMON_BEGIN(vsx_unavailable_common) + EXCEPTION_PROLOG_COMMON(0xf40, PACA_EXGEN) +#ifdef CONFIG_VSX +BEGIN_FTR_SECTION + beq 1f +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + BEGIN_FTR_SECTION_NESTED(69) + /* Test if 2 TM state bits are zero. If non-zero (ie. userspace was in + * transaction), go do TM stuff + */ + rldicl. r0, r12, (64-MSR_TS_LG), (64-2) + bne- 2f + END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) +#endif + b load_up_vsx +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +2: /* User process was in a transaction */ + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl vsx_unavailable_tm + b ret_from_except +#endif +1: +END_FTR_SECTION_IFSET(CPU_FTR_VSX) +#endif + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + addi r3,r1,STACK_FRAME_OVERHEAD + bl vsx_unavailable_exception + b ret_from_except + + +EXC_REAL_OOL(facility_unavailable, 0xf60, 0xf80) +EXC_VIRT_OOL(facility_unavailable, 0x4f60, 0x4f80, 0xf60) +TRAMP_KVM(PACA_EXGEN, 0xf60) +EXC_COMMON(facility_unavailable_common, 0xf60, facility_unavailable_exception) + + +EXC_REAL_OOL_HV(h_facility_unavailable, 0xf80, 0xfa0) +EXC_VIRT_OOL_HV(h_facility_unavailable, 0x4f80, 0x4fa0, 0xf80) +TRAMP_KVM_HV(PACA_EXGEN, 0xf80) +EXC_COMMON(h_facility_unavailable_common, 0xf80, facility_unavailable_exception) + + +EXC_REAL_NONE(0xfa0, 0x1200) +EXC_VIRT_NONE(0x4fa0, 0x5200) + +#ifdef CONFIG_CBE_RAS +EXC_REAL_HV(cbe_system_error, 0x1200, 0x1300) +EXC_VIRT_NONE(0x5200, 0x5300) +TRAMP_KVM_HV_SKIP(PACA_EXGEN, 0x1200) +EXC_COMMON(cbe_system_error_common, 0x1200, cbe_system_error_exception) +#else /* CONFIG_CBE_RAS */ +EXC_REAL_NONE(0x1200, 0x1300) +EXC_VIRT_NONE(0x5200, 0x5300) +#endif + + +EXC_REAL(instruction_breakpoint, 0x1300, 0x1400) +EXC_VIRT(instruction_breakpoint, 0x5300, 0x5400, 0x1300) +TRAMP_KVM_SKIP(PACA_EXGEN, 0x1300) +EXC_COMMON(instruction_breakpoint_common, 0x1300, instruction_breakpoint_exception) + +EXC_REAL_NONE(0x1400, 0x1500) +EXC_VIRT_NONE(0x5400, 0x5500) + +EXC_REAL_BEGIN(denorm_exception_hv, 0x1500, 0x1600) + mtspr SPRN_SPRG_HSCRATCH0,r13 + EXCEPTION_PROLOG_0(PACA_EXGEN) + EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x1500) + +#ifdef CONFIG_PPC_DENORMALISATION + mfspr r10,SPRN_HSRR1 + mfspr r11,SPRN_HSRR0 /* save HSRR0 */ + andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */ + addi r11,r11,-4 /* HSRR0 is next instruction */ + bne+ denorm_assist +#endif + + KVMTEST_PR(0x1500) + EXCEPTION_PROLOG_PSERIES_1(denorm_common, EXC_HV) +EXC_REAL_END(denorm_exception_hv, 0x1500, 0x1600) + +#ifdef CONFIG_PPC_DENORMALISATION +EXC_VIRT_BEGIN(denorm_exception, 0x5500, 0x5600) + b exc_real_0x1500_denorm_exception_hv +EXC_VIRT_END(denorm_exception, 0x5500, 0x5600) +#else +EXC_VIRT_NONE(0x5500, 0x5600) +#endif + +TRAMP_KVM_SKIP(PACA_EXGEN, 0x1500) + +#ifdef CONFIG_PPC_DENORMALISATION +TRAMP_REAL_BEGIN(denorm_assist) +BEGIN_FTR_SECTION +/* + * To denormalise we need to move a copy of the register to itself. + * For POWER6 do that here for all FP regs. + */ + mfmsr r10 + ori r10,r10,(MSR_FP|MSR_FE0|MSR_FE1) + xori r10,r10,(MSR_FE0|MSR_FE1) + mtmsrd r10 + sync + +#define FMR2(n) fmr (n), (n) ; fmr n+1, n+1 +#define FMR4(n) FMR2(n) ; FMR2(n+2) +#define FMR8(n) FMR4(n) ; FMR4(n+4) +#define FMR16(n) FMR8(n) ; FMR8(n+8) +#define FMR32(n) FMR16(n) ; FMR16(n+16) + FMR32(0) + +FTR_SECTION_ELSE +/* + * To denormalise we need to move a copy of the register to itself. + * For POWER7 do that here for the first 32 VSX registers only. + */ + mfmsr r10 + oris r10,r10,MSR_VSX@h + mtmsrd r10 + sync + +#define XVCPSGNDP2(n) XVCPSGNDP(n,n,n) ; XVCPSGNDP(n+1,n+1,n+1) +#define XVCPSGNDP4(n) XVCPSGNDP2(n) ; XVCPSGNDP2(n+2) +#define XVCPSGNDP8(n) XVCPSGNDP4(n) ; XVCPSGNDP4(n+4) +#define XVCPSGNDP16(n) XVCPSGNDP8(n) ; XVCPSGNDP8(n+8) +#define XVCPSGNDP32(n) XVCPSGNDP16(n) ; XVCPSGNDP16(n+16) + XVCPSGNDP32(0) + +ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_206) + +BEGIN_FTR_SECTION + b denorm_done +END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S) +/* + * To denormalise we need to move a copy of the register to itself. + * For POWER8 we need to do that for all 64 VSX registers + */ + XVCPSGNDP32(32) +denorm_done: + mtspr SPRN_HSRR0,r11 + mtcrf 0x80,r9 + ld r9,PACA_EXGEN+EX_R9(r13) + RESTORE_PPR_PACA(PACA_EXGEN, r10) +BEGIN_FTR_SECTION + ld r10,PACA_EXGEN+EX_CFAR(r13) + mtspr SPRN_CFAR,r10 +END_FTR_SECTION_IFSET(CPU_FTR_CFAR) + ld r10,PACA_EXGEN+EX_R10(r13) + ld r11,PACA_EXGEN+EX_R11(r13) + ld r12,PACA_EXGEN+EX_R12(r13) + ld r13,PACA_EXGEN+EX_R13(r13) + HRFID + b . +#endif + +EXC_COMMON_HV(denorm_common, 0x1500, unknown_exception) + + +#ifdef CONFIG_CBE_RAS +EXC_REAL_HV(cbe_maintenance, 0x1600, 0x1700) +EXC_VIRT_NONE(0x5600, 0x5700) +TRAMP_KVM_HV_SKIP(PACA_EXGEN, 0x1600) +EXC_COMMON(cbe_maintenance_common, 0x1600, cbe_maintenance_exception) +#else /* CONFIG_CBE_RAS */ +EXC_REAL_NONE(0x1600, 0x1700) +EXC_VIRT_NONE(0x5600, 0x5700) +#endif + + +EXC_REAL(altivec_assist, 0x1700, 0x1800) +EXC_VIRT(altivec_assist, 0x5700, 0x5800, 0x1700) +TRAMP_KVM(PACA_EXGEN, 0x1700) +#ifdef CONFIG_ALTIVEC +EXC_COMMON(altivec_assist_common, 0x1700, altivec_assist_exception) +#else +EXC_COMMON(altivec_assist_common, 0x1700, unknown_exception) +#endif + + +#ifdef CONFIG_CBE_RAS +EXC_REAL_HV(cbe_thermal, 0x1800, 0x1900) +EXC_VIRT_NONE(0x5800, 0x5900) +TRAMP_KVM_HV_SKIP(PACA_EXGEN, 0x1800) +EXC_COMMON(cbe_thermal_common, 0x1800, cbe_thermal_exception) +#else /* CONFIG_CBE_RAS */ +EXC_REAL_NONE(0x1800, 0x1900) +EXC_VIRT_NONE(0x5800, 0x5900) +#endif + + +/* + * An interrupt came in while soft-disabled. We set paca->irq_happened, then: + * - If it was a decrementer interrupt, we bump the dec to max and and return. + * - If it was a doorbell we return immediately since doorbells are edge + * triggered and won't automatically refire. + * - If it was a HMI we return immediately since we handled it in realmode + * and it won't refire. + * - else we hard disable and return. + * This is called with r10 containing the value to OR to the paca field. + */ +#define MASKED_INTERRUPT(_H) \ +masked_##_H##interrupt: \ + std r11,PACA_EXGEN+EX_R11(r13); \ + lbz r11,PACAIRQHAPPENED(r13); \ + or r11,r11,r10; \ + stb r11,PACAIRQHAPPENED(r13); \ + cmpwi r10,PACA_IRQ_DEC; \ + bne 1f; \ + lis r10,0x7fff; \ + ori r10,r10,0xffff; \ + mtspr SPRN_DEC,r10; \ + b 2f; \ +1: cmpwi r10,PACA_IRQ_DBELL; \ + beq 2f; \ + cmpwi r10,PACA_IRQ_HMI; \ + beq 2f; \ + mfspr r10,SPRN_##_H##SRR1; \ + rldicl r10,r10,48,1; /* clear MSR_EE */ \ + rotldi r10,r10,16; \ + mtspr SPRN_##_H##SRR1,r10; \ +2: mtcrf 0x80,r9; \ + ld r9,PACA_EXGEN+EX_R9(r13); \ + ld r10,PACA_EXGEN+EX_R10(r13); \ + ld r11,PACA_EXGEN+EX_R11(r13); \ + GET_SCRATCH0(r13); \ + ##_H##rfid; \ + b . + +/* + * Real mode exceptions actually use this too, but alternate + * instruction code patches (which end up in the common .text area) + * cannot reach these if they are put there. + */ +USE_FIXED_SECTION(virt_trampolines) + MASKED_INTERRUPT() + MASKED_INTERRUPT(H) + +#ifdef CONFIG_KVM_BOOK3S_64_HANDLER +TRAMP_REAL_BEGIN(kvmppc_skip_interrupt) + /* + * Here all GPRs are unchanged from when the interrupt happened + * except for r13, which is saved in SPRG_SCRATCH0. + */ + mfspr r13, SPRN_SRR0 + addi r13, r13, 4 + mtspr SPRN_SRR0, r13 + GET_SCRATCH0(r13) + rfid + b . + +TRAMP_REAL_BEGIN(kvmppc_skip_Hinterrupt) + /* + * Here all GPRs are unchanged from when the interrupt happened + * except for r13, which is saved in SPRG_SCRATCH0. + */ + mfspr r13, SPRN_HSRR0 + addi r13, r13, 4 + mtspr SPRN_HSRR0, r13 + GET_SCRATCH0(r13) + hrfid + b . +#endif + +/* + * Ensure that any handlers that get invoked from the exception prologs + * above are below the first 64KB (0x10000) of the kernel image because + * the prologs assemble the addresses of these handlers using the + * LOAD_HANDLER macro, which uses an ori instruction. + */ + +/*** Common interrupt handlers ***/ + + + /* + * Relocation-on interrupts: A subset of the interrupts can be delivered + * with IR=1/DR=1, if AIL==2 and MSR.HV won't be changed by delivering + * it. Addresses are the same as the original interrupt addresses, but + * offset by 0xc000000000004000. + * It's impossible to receive interrupts below 0x300 via this mechanism. + * KVM: None of these traps are from the guest ; anything that escalated + * to HV=1 from HV=0 is delivered via real mode handlers. + */ + + /* + * This uses the standard macro, since the original 0x300 vector + * only has extra guff for STAB-based processors -- which never + * come here. + */ + +EXC_COMMON_BEGIN(ppc64_runlatch_on_trampoline) + b __ppc64_runlatch_on + +USE_FIXED_SECTION(virt_trampolines) + /* + * The __end_interrupts marker must be past the out-of-line (OOL) + * handlers, so that they are copied to real address 0x100 when running + * a relocatable kernel. This ensures they can be reached from the short + * trampoline handlers (like 0x4f00, 0x4f20, etc.) which branch + * directly, without using LOAD_HANDLER(). + */ + .align 7 + .globl __end_interrupts +__end_interrupts: +DEFINE_FIXED_SYMBOL(__end_interrupts) #ifdef CONFIG_PPC_970_NAP -power4_fixup_nap: +TRAMP_REAL_BEGIN(power4_fixup_nap) andc r9,r9,r10 std r9,TI_LOCAL_FLAGS(r11) ld r10,_LINK(r1) /* make idle task do the */ @@ -1480,6 +1385,13 @@ power4_fixup_nap: blr #endif +CLOSE_FIXED_SECTION(real_vectors); +CLOSE_FIXED_SECTION(real_trampolines); +CLOSE_FIXED_SECTION(virt_vectors); +CLOSE_FIXED_SECTION(virt_trampolines); + +USE_TEXT_SECTION() + /* * Hash table stuff */ @@ -1625,3 +1537,39 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) 1: addi r3,r1,STACK_FRAME_OVERHEAD bl kernel_bad_stack b 1b + +/* + * Called from arch_local_irq_enable when an interrupt needs + * to be resent. r3 contains 0x500, 0x900, 0xa00 or 0xe80 to indicate + * which kind of interrupt. MSR:EE is already off. We generate a + * stackframe like if a real interrupt had happened. + * + * Note: While MSR:EE is off, we need to make sure that _MSR + * in the generated frame has EE set to 1 or the exception + * handler will not properly re-enable them. + */ +_GLOBAL(__replay_interrupt) + /* We are going to jump to the exception common code which + * will retrieve various register values from the PACA which + * we don't give a damn about, so we don't bother storing them. + */ + mfmsr r12 + mflr r11 + mfcr r9 + ori r12,r12,MSR_EE + cmpwi r3,0x900 + beq decrementer_common + cmpwi r3,0x500 + beq hardware_interrupt_common +BEGIN_FTR_SECTION + cmpwi r3,0xe80 + beq h_doorbell_common + cmpwi r3,0xea0 + beq h_virt_irq_common + cmpwi r3,0xe60 + beq hmi_exception_common +FTR_SECTION_ELSE + cmpwi r3,0xa00 + beq doorbell_super_common +ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE) + blr diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c index b3a663333d36..963918e0974a 100644 --- a/arch/powerpc/kernel/fadump.c +++ b/arch/powerpc/kernel/fadump.c @@ -778,7 +778,11 @@ static int fadump_init_elfcore_header(char *bufp) elf->e_entry = 0; elf->e_phoff = sizeof(struct elfhdr); elf->e_shoff = 0; - elf->e_flags = ELF_CORE_EFLAGS; +#if defined(_CALL_ELF) + elf->e_flags = _CALL_ELF; +#else + elf->e_flags = 0; +#endif elf->e_ehsize = sizeof(struct elfhdr); elf->e_phentsize = sizeof(struct elf_phdr); elf->e_phnum = 0; @@ -1104,7 +1108,9 @@ static ssize_t fadump_release_memory_store(struct kobject *kobj, * Take away the '/proc/vmcore'. We are releasing the dump * memory, hence it will not be valid anymore. */ +#ifdef CONFIG_PROC_VMCORE vmcore_cleanup(); +#endif fadump_invalidate_release_mem(); } else diff --git a/arch/powerpc/kernel/fpu.S b/arch/powerpc/kernel/fpu.S index 15da2b5df85e..08d14b096eb9 100644 --- a/arch/powerpc/kernel/fpu.S +++ b/arch/powerpc/kernel/fpu.S @@ -50,32 +50,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX); \ #define REST_32FPVSRS(n,c,base) __REST_32FPVSRS(n,__REG_##c,__REG_##base) #define SAVE_32FPVSRS(n,c,base) __SAVE_32FPVSRS(n,__REG_##c,__REG_##base) -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -/* void do_load_up_transact_fpu(struct thread_struct *thread) - * - * This is similar to load_up_fpu but for the transactional version of the FP - * register set. It doesn't mess with the task MSR or valid flags. - * Furthermore, we don't do lazy FP with TM currently. - */ -_GLOBAL(do_load_up_transact_fpu) - mfmsr r6 - ori r5,r6,MSR_FP -#ifdef CONFIG_VSX -BEGIN_FTR_SECTION - oris r5,r5,MSR_VSX@h -END_FTR_SECTION_IFSET(CPU_FTR_VSX) -#endif - SYNC - MTMSRD(r5) - - addi r7,r3,THREAD_TRANSACT_FPSTATE - lfd fr0,FPSTATE_FPSCR(r7) - MTFSF_L(fr0) - REST_32FPVSRS(0, R4, R7) - - blr -#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ - /* * Load state from memory into FP registers including FPSCR. * Assumes the caller has enabled FP in the MSR. diff --git a/arch/powerpc/kernel/head_32.S b/arch/powerpc/kernel/head_32.S index dc0488b6f6e1..a3f821eb7e9a 100644 --- a/arch/powerpc/kernel/head_32.S +++ b/arch/powerpc/kernel/head_32.S @@ -266,7 +266,6 @@ __secondary_hold_acknowledge: #define EXCEPTION_PROLOG_2 \ - CLR_TOP32(r11); \ stw r10,_CCR(r11); /* save registers */ \ stw r12,GPR12(r11); \ stw r9,GPR9(r11); \ @@ -862,7 +861,6 @@ __secondary_start: /* ptr to phys current thread */ tophys(r4,r2) addi r4,r4,THREAD /* phys address of our thread_struct */ - CLR_TOP32(r4) mtspr SPRN_SPRG_THREAD,r4 li r3,0 mtspr SPRN_SPRG_RTAS,r3 /* 0 => not in RTAS */ @@ -949,7 +947,6 @@ start_here: /* ptr to phys current thread */ tophys(r4,r2) addi r4,r4,THREAD /* init task's THREAD */ - CLR_TOP32(r4) mtspr SPRN_SPRG_THREAD,r4 li r3,0 mtspr SPRN_SPRG_RTAS,r3 /* 0 => not in RTAS */ diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S index f765b0434731..79da0641bae2 100644 --- a/arch/powerpc/kernel/head_64.S +++ b/arch/powerpc/kernel/head_64.S @@ -28,6 +28,7 @@ #include <asm/page.h> #include <asm/mmu.h> #include <asm/ppc_asm.h> +#include <asm/head-64.h> #include <asm/asm-offsets.h> #include <asm/bug.h> #include <asm/cputable.h> @@ -65,9 +66,14 @@ * 2. The kernel is entered at __start */ - .text - .globl _stext -_stext: +OPEN_FIXED_SECTION(first_256B, 0x0, 0x100) +USE_FIXED_SECTION(first_256B) + /* + * Offsets are relative from the start of fixed section, and + * first_256B starts at 0. Offsets are a bit easier to use here + * than the fixed section entry macros. + */ + . = 0x0 _GLOBAL(__start) /* NOP this out unconditionally */ BEGIN_FTR_SECTION @@ -104,6 +110,7 @@ __secondary_hold_acknowledge: . = 0x5c .globl __run_at_load __run_at_load: +DEFINE_FIXED_SYMBOL(__run_at_load) .long 0x72756e30 /* "run0" -- relocate to 0 by default */ #endif @@ -133,7 +140,7 @@ __secondary_hold: /* Tell the master cpu we're here */ /* Relocation is off & we are located at an address less */ /* than 0x100, so only need to grab low order offset. */ - std r24,__secondary_hold_acknowledge-_stext(0) + std r24,(ABS_ADDR(__secondary_hold_acknowledge))(0) sync li r26,0 @@ -141,7 +148,7 @@ __secondary_hold: tovirt(r26,r26) #endif /* All secondary cpus wait here until told to start. */ -100: ld r12,__secondary_hold_spinloop-_stext(r26) +100: ld r12,(ABS_ADDR(__secondary_hold_spinloop))(r26) cmpdi 0,r12,0 beq 100b @@ -166,12 +173,13 @@ __secondary_hold: #else BUG_OPCODE #endif +CLOSE_FIXED_SECTION(first_256B) /* This value is used to mark exception frames on the stack. */ .section ".toc","aw" exception_marker: .tc ID_72656773_68657265[TC],0x7265677368657265 - .text + .previous /* * On server, we include the exception vectors code here as it @@ -180,8 +188,12 @@ exception_marker: */ #ifdef CONFIG_PPC_BOOK3S #include "exceptions-64s.S" +#else +OPEN_TEXT_SECTION(0x100) #endif +USE_TEXT_SECTION() + #ifdef CONFIG_PPC_BOOK3E /* * The booting_thread_hwid holds the thread id we want to boot in cpu @@ -558,7 +570,7 @@ __after_prom_start: #if defined(CONFIG_PPC_BOOK3E) tovirt(r26,r26) /* on booke, we already run at PAGE_OFFSET */ #endif - lwz r7,__run_at_load-_stext(r26) + lwz r7,(FIXED_SYMBOL_ABS_ADDR(__run_at_load))(r26) #if defined(CONFIG_PPC_BOOK3E) tophys(r26,r26) #endif @@ -601,7 +613,7 @@ __after_prom_start: #if defined(CONFIG_PPC_BOOK3E) tovirt(r26,r26) /* on booke, we already run at PAGE_OFFSET */ #endif - lwz r7,__run_at_load-_stext(r26) + lwz r7,(FIXED_SYMBOL_ABS_ADDR(__run_at_load))(r26) cmplwi cr0,r7,1 bne 3f @@ -611,28 +623,35 @@ __after_prom_start: sub r5,r5,r11 #else /* just copy interrupts */ - LOAD_REG_IMMEDIATE(r5, __end_interrupts - _stext) + LOAD_REG_IMMEDIATE(r5, FIXED_SYMBOL_ABS_ADDR(__end_interrupts)) #endif b 5f 3: #endif - lis r5,(copy_to_here - _stext)@ha - addi r5,r5,(copy_to_here - _stext)@l /* # bytes of memory to copy */ + /* # bytes of memory to copy */ + lis r5,(ABS_ADDR(copy_to_here))@ha + addi r5,r5,(ABS_ADDR(copy_to_here))@l bl copy_and_flush /* copy the first n bytes */ /* this includes the code being */ /* executed here. */ - addis r8,r3,(4f - _stext)@ha /* Jump to the copy of this code */ - addi r12,r8,(4f - _stext)@l /* that we just made */ + /* Jump to the copy of this code that we just made */ + addis r8,r3,(ABS_ADDR(4f))@ha + addi r12,r8,(ABS_ADDR(4f))@l mtctr r12 bctr .balign 8 -p_end: .llong _end - _stext +p_end: .llong _end - copy_to_here -4: /* Now copy the rest of the kernel up to _end */ - addis r5,r26,(p_end - _stext)@ha - ld r5,(p_end - _stext)@l(r5) /* get _end */ +4: + /* + * Now copy the rest of the kernel up to _end, add + * _end - copy_to_here to the copy limit and run again. + */ + addis r8,r26,(ABS_ADDR(p_end))@ha + ld r8,(ABS_ADDR(p_end))@l(r8) + add r5,r5,r8 5: bl copy_and_flush /* copy the rest */ 9: b start_here_multiplatform diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index 43ddaae42baf..3a185c51ce8f 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -151,7 +151,6 @@ turn_on_mmu: #define EXCEPTION_PROLOG_2 \ - CLR_TOP32(r11); \ stw r10,_CCR(r11); /* save registers */ \ stw r12,GPR12(r11); \ stw r9,GPR9(r11); \ diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c index aec9a1b1d25b..9781c69eae57 100644 --- a/arch/powerpc/kernel/hw_breakpoint.c +++ b/arch/powerpc/kernel/hw_breakpoint.c @@ -206,7 +206,7 @@ void thread_change_pc(struct task_struct *tsk, struct pt_regs *regs) /* * Handle debug exception notifications. */ -int __kprobes hw_breakpoint_handler(struct die_args *args) +int hw_breakpoint_handler(struct die_args *args) { int rc = NOTIFY_STOP; struct perf_event *bp; @@ -290,11 +290,12 @@ out: rcu_read_unlock(); return rc; } +NOKPROBE_SYMBOL(hw_breakpoint_handler); /* * Handle single-step exceptions following a DABR hit. */ -static int __kprobes single_step_dabr_instruction(struct die_args *args) +static int single_step_dabr_instruction(struct die_args *args) { struct pt_regs *regs = args->regs; struct perf_event *bp = NULL; @@ -329,11 +330,12 @@ static int __kprobes single_step_dabr_instruction(struct die_args *args) return NOTIFY_STOP; } +NOKPROBE_SYMBOL(single_step_dabr_instruction); /* * Handle debug exception notifications. */ -int __kprobes hw_breakpoint_exceptions_notify( +int hw_breakpoint_exceptions_notify( struct notifier_block *unused, unsigned long val, void *data) { int ret = NOTIFY_DONE; @@ -349,6 +351,7 @@ int __kprobes hw_breakpoint_exceptions_notify( return ret; } +NOKPROBE_SYMBOL(hw_breakpoint_exceptions_notify); /* * Release the user breakpoints used by ptrace diff --git a/arch/powerpc/kernel/ibmebus.c b/arch/powerpc/kernel/ibmebus.c index c1ca9282f4a0..6ca9a2ffaac7 100644 --- a/arch/powerpc/kernel/ibmebus.c +++ b/arch/powerpc/kernel/ibmebus.c @@ -227,7 +227,7 @@ int ibmebus_request_irq(u32 ist, irq_handler_t handler, { unsigned int irq = irq_create_mapping(NULL, ist); - if (irq == NO_IRQ) + if (!irq) return -EINVAL; return request_irq(irq, handler, irq_flags, devname, dev_id); diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c index 08887cf2b20e..3c05c311e35e 100644 --- a/arch/powerpc/kernel/irq.c +++ b/arch/powerpc/kernel/irq.c @@ -67,6 +67,7 @@ #include <asm/smp.h> #include <asm/debug.h> #include <asm/livepatch.h> +#include <asm/asm-prototypes.h> #ifdef CONFIG_PPC64 #include <asm/paca.h> @@ -156,6 +157,15 @@ notrace unsigned int __check_irq_replay(void) } /* + * Check if an hypervisor Maintenance interrupt happened. + * This is a higher priority interrupt than the others, so + * replay it first. + */ + local_paca->irq_happened &= ~PACA_IRQ_HMI; + if (happened & PACA_IRQ_HMI) + return 0xe60; + + /* * We may have missed a decrementer interrupt. We check the * decrementer itself rather than the paca irq_happened field * in case we also had a rollover while hard disabled @@ -190,11 +200,6 @@ notrace unsigned int __check_irq_replay(void) } #endif /* CONFIG_PPC_BOOK3E */ - /* Check if an hypervisor Maintenance interrupt happened */ - local_paca->irq_happened &= ~PACA_IRQ_HMI; - if (happened & PACA_IRQ_HMI) - return 0xe60; - /* There should be nothing left ! */ BUG_ON(local_paca->irq_happened != 0); @@ -514,7 +519,7 @@ void __do_irq(struct pt_regs *regs) may_hard_irq_enable(); /* And finally process it */ - if (unlikely(irq == NO_IRQ)) + if (unlikely(!irq)) __this_cpu_inc(irq_stat.spurious_irqs); else generic_handle_irq(irq); diff --git a/arch/powerpc/kernel/legacy_serial.c b/arch/powerpc/kernel/legacy_serial.c index 7b750c4ed5c7..bc525ea0dc09 100644 --- a/arch/powerpc/kernel/legacy_serial.c +++ b/arch/powerpc/kernel/legacy_serial.c @@ -193,10 +193,10 @@ static int __init add_legacy_soc_port(struct device_node *np, */ if (tsi && !strcmp(tsi->type, "tsi-bridge")) return add_legacy_port(np, -1, UPIO_TSI, addr, addr, - NO_IRQ, legacy_port_flags, 0); + 0, legacy_port_flags, 0); else return add_legacy_port(np, -1, UPIO_MEM, addr, addr, - NO_IRQ, legacy_port_flags, 0); + 0, legacy_port_flags, 0); } static int __init add_legacy_isa_port(struct device_node *np, @@ -242,7 +242,7 @@ static int __init add_legacy_isa_port(struct device_node *np, /* Add port, irq will be dealt with later */ return add_legacy_port(np, index, UPIO_PORT, be32_to_cpu(reg[1]), - taddr, NO_IRQ, legacy_port_flags, 0); + taddr, 0, legacy_port_flags, 0); } @@ -314,7 +314,7 @@ static int __init add_legacy_pci_port(struct device_node *np, /* Add port, irq will be dealt with later. We passed a translated * IO port value. It will be fixed up later along with the irq */ - return add_legacy_port(np, index, iotype, base, addr, NO_IRQ, + return add_legacy_port(np, index, iotype, base, addr, 0, legacy_port_flags, np != pci_dev); } #endif @@ -462,14 +462,14 @@ static void __init fixup_port_irq(int index, DBG("fixup_port_irq(%d)\n", index); virq = irq_of_parse_and_map(np, 0); - if (virq == NO_IRQ && legacy_serial_infos[index].irq_check_parent) { + if (!virq && legacy_serial_infos[index].irq_check_parent) { np = of_get_parent(np); if (np == NULL) return; virq = irq_of_parse_and_map(np, 0); of_node_put(np); } - if (virq == NO_IRQ) + if (!virq) return; port->irq = virq; @@ -543,7 +543,7 @@ static int __init serial_dev_init(void) struct plat_serial8250_port *port = &legacy_serial_ports[i]; struct device_node *np = legacy_serial_infos[i].np; - if (port->irq == NO_IRQ) + if (!port->irq) fixup_port_irq(i, np, port); if (port->iotype == UPIO_PORT) fixup_port_pio(i, np, port); diff --git a/arch/powerpc/kernel/machine_kexec_64.c b/arch/powerpc/kernel/machine_kexec_64.c index 4c780a342282..a205fa3d9bf3 100644 --- a/arch/powerpc/kernel/machine_kexec_64.c +++ b/arch/powerpc/kernel/machine_kexec_64.c @@ -23,6 +23,7 @@ #include <asm/current.h> #include <asm/machdep.h> #include <asm/cacheflush.h> +#include <asm/firmware.h> #include <asm/paca.h> #include <asm/mmu.h> #include <asm/sections.h> /* _end */ @@ -31,21 +32,6 @@ #include <asm/hw_breakpoint.h> #include <asm/asm-prototypes.h> -#ifdef CONFIG_PPC_BOOK3E -int default_machine_kexec_prepare(struct kimage *image) -{ - int i; - /* - * Since we use the kernel fault handlers and paging code to - * handle the virtual mode, we must make sure no destination - * overlaps kernel static data or bss. - */ - for (i = 0; i < image->nr_segments; i++) - if (image->segment[i].mem < __pa(_end)) - return -ETXTBSY; - return 0; -} -#else int default_machine_kexec_prepare(struct kimage *image) { int i; @@ -55,9 +41,6 @@ int default_machine_kexec_prepare(struct kimage *image) const unsigned long *basep; const unsigned int *sizep; - if (!mmu_hash_ops.hpte_clear_all) - return -ENOENT; - /* * Since we use the kernel fault handlers and paging code to * handle the virtual mode, we must make sure no destination @@ -67,31 +50,6 @@ int default_machine_kexec_prepare(struct kimage *image) if (image->segment[i].mem < __pa(_end)) return -ETXTBSY; - /* - * For non-LPAR, we absolutely can not overwrite the mmu hash - * table, since we are still using the bolted entries in it to - * do the copy. Check that here. - * - * It is safe if the end is below the start of the blocked - * region (end <= low), or if the beginning is after the - * end of the blocked region (begin >= high). Use the - * boolean identity !(a || b) === (!a && !b). - */ -#ifdef CONFIG_PPC_STD_MMU_64 - if (htab_address) { - low = __pa(htab_address); - high = low + htab_size_bytes; - - for (i = 0; i < image->nr_segments; i++) { - begin = image->segment[i].mem; - end = begin + image->segment[i].memsz; - - if ((begin < high) && (end > low)) - return -ETXTBSY; - } - } -#endif /* CONFIG_PPC_STD_MMU_64 */ - /* We also should not overwrite the tce tables */ for_each_node_by_type(node, "pci") { basep = of_get_property(node, "linux,tce-base", NULL); @@ -113,7 +71,6 @@ int default_machine_kexec_prepare(struct kimage *image) return 0; } -#endif /* !CONFIG_PPC_BOOK3E */ static void copy_segments(unsigned long ind) { @@ -332,11 +289,14 @@ struct paca_struct kexec_paca; /* Our assembly helper, in misc_64.S */ extern void kexec_sequence(void *newstack, unsigned long start, void *image, void *control, - void (*clear_all)(void)) __noreturn; + void (*clear_all)(void), + bool copy_with_mmu_off) __noreturn; /* too late to fail here */ void default_machine_kexec(struct kimage *image) { + bool copy_with_mmu_off; + /* prepare control code if any */ /* @@ -374,18 +334,29 @@ void default_machine_kexec(struct kimage *image) /* XXX: If anyone does 'dynamic lppacas' this will also need to be * switched to a static version! */ + /* + * On Book3S, the copy must happen with the MMU off if we are either + * using Radix page tables or we are not in an LPAR since we can + * overwrite the page tables while copying. + * + * In an LPAR, we keep the MMU on otherwise we can't access beyond + * the RMA. On BookE there is no real MMU off mode, so we have to + * keep it enabled as well (but then we have bolted TLB entries). + */ +#ifdef CONFIG_PPC_BOOK3E + copy_with_mmu_off = false; +#else + copy_with_mmu_off = radix_enabled() || + !(firmware_has_feature(FW_FEATURE_LPAR) || + firmware_has_feature(FW_FEATURE_PS3_LV1)); +#endif /* Some things are best done in assembly. Finding globals with * a toc is easier in C, so pass in what we can. */ kexec_sequence(&kexec_stack, image->start, image, - page_address(image->control_code_page), -#ifdef CONFIG_PPC_STD_MMU - mmu_hash_ops.hpte_clear_all -#else - NULL -#endif - ); + page_address(image->control_code_page), + mmu_cleanup_all, copy_with_mmu_off); /* NOTREACHED */ } diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S index d9c912b6e632..03756ffdcd71 100644 --- a/arch/powerpc/kernel/misc_32.S +++ b/arch/powerpc/kernel/misc_32.S @@ -328,7 +328,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_UNIFIED_ID_CACHE) * * flush_icache_range(unsigned long start, unsigned long stop) */ -_KPROBE(flush_icache_range) +_GLOBAL(flush_icache_range) BEGIN_FTR_SECTION PURGE_PREFETCHED_INS blr /* for 601, do nothing */ @@ -358,6 +358,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_COHERENT_ICACHE) sync /* additional sync needed on g4 */ isync blr +_ASM_NOKPROBE_SYMBOL(flush_icache_range) + /* * Flush a particular page from the data cache to RAM. * Note: this is necessary because the instruction cache does *not* diff --git a/arch/powerpc/kernel/misc_64.S b/arch/powerpc/kernel/misc_64.S index cb195157b318..9f0bed214bcb 100644 --- a/arch/powerpc/kernel/misc_64.S +++ b/arch/powerpc/kernel/misc_64.S @@ -66,7 +66,7 @@ PPC64_CACHES: * flush all bytes from start through stop-1 inclusive */ -_KPROBE(flush_icache_range) +_GLOBAL(flush_icache_range) BEGIN_FTR_SECTION PURGE_PREFETCHED_INS blr @@ -109,7 +109,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_COHERENT_ICACHE) bdnz 2b isync blr - .previous .text +_ASM_NOKPROBE_SYMBOL(flush_icache_range) + /* * Like above, but only do the D-cache. * @@ -591,7 +592,8 @@ real_mode: /* assume normal blr return */ #endif /* - * kexec_sequence(newstack, start, image, control, clear_all()) + * kexec_sequence(newstack, start, image, control, clear_all(), + copy_with_mmu_off) * * does the grungy work with stack switching and real mode switches * also does simple calls to other code @@ -627,7 +629,7 @@ _GLOBAL(kexec_sequence) mr r29,r5 /* image (virt) */ mr r28,r6 /* control, unused */ mr r27,r7 /* clear_all() fn desc */ - mr r26,r8 /* spare */ + mr r26,r8 /* copy_with_mmu_off */ lhz r25,PACAHWCPUID(r13) /* get our phys cpu from paca */ /* disable interrupts, we are overwriting kernel data next */ @@ -639,15 +641,24 @@ _GLOBAL(kexec_sequence) mtmsrd r3,1 #endif + /* We need to turn the MMU off unless we are in hash mode + * under a hypervisor + */ + cmpdi r26,0 + beq 1f + bl real_mode +1: /* copy dest pages, flush whole dest image */ mr r3,r29 bl kexec_copy_flush /* (image) */ - /* turn off mmu */ + /* turn off mmu now if not done earlier */ + cmpdi r26,0 + bne 1f bl real_mode /* copy 0x100 bytes starting at start to 0 */ - li r3,0 +1: li r3,0 mr r4,r30 /* start, aka phys mem offset */ li r5,0x100 li r6,0 @@ -659,7 +670,9 @@ _GLOBAL(kexec_sequence) li r6,1 stw r6,kexec_flag-1b(5) -#ifndef CONFIG_PPC_BOOK3E + cmpdi r27,0 + beq 1f + /* clear out hardware hash page table and tlb */ #ifdef PPC64_ELF_ABI_v1 ld r12,0(r27) /* deref function descriptor */ @@ -668,7 +681,6 @@ _GLOBAL(kexec_sequence) #endif mtctr r12 bctrl /* mmu_hash_ops.hpte_clear_all(void); */ -#endif /* !CONFIG_PPC_BOOK3E */ /* * kexec image calling is: @@ -695,7 +707,7 @@ _GLOBAL(kexec_sequence) * are the boot cpu ????? * other device tree differences (prop sizes, va vs pa, etc)... */ - mr r3,r25 # my phys cpu +1: mr r3,r25 # my phys cpu mr r4,r30 # start, aka phys mem offset mtlr 4 li r5,0 diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index d1f1b35bf0c7..30b89d5cbb03 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -27,7 +27,7 @@ #include <linux/sort.h> #include <asm/setup.h> -LIST_HEAD(module_bug_list); +static LIST_HEAD(module_bug_list); static const Elf_Shdr *find_section(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, diff --git a/arch/powerpc/kernel/nvram_64.c b/arch/powerpc/kernel/nvram_64.c index 64174bf95611..34d2c595de23 100644 --- a/arch/powerpc/kernel/nvram_64.c +++ b/arch/powerpc/kernel/nvram_64.c @@ -542,9 +542,9 @@ static ssize_t nvram_pstore_read(u64 *id, enum pstore_type_id *type, time->tv_nsec = 0; } *buf = kmemdup(buff + hdr_size, length, GFP_KERNEL); + kfree(buff); if (*buf == NULL) return -ENOMEM; - kfree(buff); *ecc_notice_size = 0; if (err_type == ERR_TYPE_KERNEL_PANIC_GZ) @@ -851,7 +851,7 @@ static long dev_nvram_ioctl(struct file *file, unsigned int cmd, } } -const struct file_operations nvram_fops = { +static const struct file_operations nvram_fops = { .owner = THIS_MODULE, .llseek = dev_nvram_llseek, .read = dev_nvram_read, @@ -956,7 +956,7 @@ int __init nvram_remove_partition(const char *name, int sig, /* Make partition a free partition */ part->header.signature = NVRAM_SIG_FREE; - strncpy(part->header.name, "wwwwwwwwwwww", 12); + memset(part->header.name, 'w', 12); part->header.checksum = nvram_checksum(&part->header); rc = nvram_write_header(part); if (rc <= 0) { @@ -974,8 +974,8 @@ int __init nvram_remove_partition(const char *name, int sig, } if (prev) { prev->header.length += part->header.length; - prev->header.checksum = nvram_checksum(&part->header); - rc = nvram_write_header(part); + prev->header.checksum = nvram_checksum(&prev->header); + rc = nvram_write_header(prev); if (rc <= 0) { printk(KERN_ERR "nvram_remove_partition: nvram_write failed (%d)\n", rc); return rc; diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c index e58908066b0e..95d3769a2e26 100644 --- a/arch/powerpc/kernel/pci-common.c +++ b/arch/powerpc/kernel/pci-common.c @@ -360,7 +360,7 @@ static int pci_read_irq_line(struct pci_dev *pci_dev) line, pin); virq = irq_create_mapping(NULL, line); - if (virq != NO_IRQ) + if (virq) irq_set_irq_type(virq, IRQ_TYPE_LEVEL_LOW); } else { pr_debug(" Got one, spec %d cells (0x%08x 0x%08x...) on %s\n", @@ -369,7 +369,8 @@ static int pci_read_irq_line(struct pci_dev *pci_dev) virq = irq_create_of_mapping(&oirq); } - if(virq == NO_IRQ) { + + if (!virq) { pr_debug(" Failed to map !\n"); return -1; } diff --git a/arch/powerpc/kernel/pci_of_scan.c b/arch/powerpc/kernel/pci_of_scan.c index 526ac6750e4d..ea3d98115b88 100644 --- a/arch/powerpc/kernel/pci_of_scan.c +++ b/arch/powerpc/kernel/pci_of_scan.c @@ -178,7 +178,7 @@ struct pci_dev *of_create_pci_dev(struct device_node *node, dev->hdr_type = PCI_HEADER_TYPE_NORMAL; dev->rom_base_reg = PCI_ROM_ADDRESS; /* Maybe do a default OF mapping here */ - dev->irq = NO_IRQ; + dev->irq = 0; } of_pci_parse_addrs(node, dev); diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index 9ee2623e0f67..9e7c10fe205f 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -59,6 +59,7 @@ #include <asm/exec.h> #include <asm/livepatch.h> #include <asm/cpu_has_feature.h> +#include <asm/asm-prototypes.h> #include <linux/kprobes.h> #include <linux/kdebug.h> @@ -88,7 +89,13 @@ static void check_if_tm_restore_required(struct task_struct *tsk) set_thread_flag(TIF_RESTORE_TM); } } + +static inline bool msr_tm_active(unsigned long msr) +{ + return MSR_TM_ACTIVE(msr); +} #else +static inline bool msr_tm_active(unsigned long msr) { return false; } static inline void check_if_tm_restore_required(struct task_struct *tsk) { } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ @@ -104,7 +111,7 @@ static int __init enable_strict_msr_control(char *str) } early_param("ppc_strict_facility_enable", enable_strict_msr_control); -void msr_check_and_set(unsigned long bits) +unsigned long msr_check_and_set(unsigned long bits) { unsigned long oldmsr = mfmsr(); unsigned long newmsr; @@ -118,6 +125,8 @@ void msr_check_and_set(unsigned long bits) if (oldmsr != newmsr) mtmsr_isync(newmsr); + + return newmsr; } void __msr_check_and_clear(unsigned long bits) @@ -196,19 +205,30 @@ EXPORT_SYMBOL_GPL(flush_fp_to_thread); void enable_kernel_fp(void) { + unsigned long cpumsr; + WARN_ON(preemptible()); - msr_check_and_set(MSR_FP); + cpumsr = msr_check_and_set(MSR_FP); if (current->thread.regs && (current->thread.regs->msr & MSR_FP)) { check_if_tm_restore_required(current); + /* + * If a thread has already been reclaimed then the + * checkpointed registers are on the CPU but have definitely + * been saved by the reclaim code. Don't need to and *cannot* + * giveup as this would save to the 'live' structure not the + * checkpointed structure. + */ + if(!msr_tm_active(cpumsr) && msr_tm_active(current->thread.regs->msr)) + return; __giveup_fpu(current); } } EXPORT_SYMBOL(enable_kernel_fp); static int restore_fp(struct task_struct *tsk) { - if (tsk->thread.load_fp) { + if (tsk->thread.load_fp || msr_tm_active(tsk->thread.regs->msr)) { load_fp_state(¤t->thread.fp_state); current->thread.load_fp++; return 1; @@ -248,12 +268,23 @@ EXPORT_SYMBOL(giveup_altivec); void enable_kernel_altivec(void) { + unsigned long cpumsr; + WARN_ON(preemptible()); - msr_check_and_set(MSR_VEC); + cpumsr = msr_check_and_set(MSR_VEC); if (current->thread.regs && (current->thread.regs->msr & MSR_VEC)) { check_if_tm_restore_required(current); + /* + * If a thread has already been reclaimed then the + * checkpointed registers are on the CPU but have definitely + * been saved by the reclaim code. Don't need to and *cannot* + * giveup as this would save to the 'live' structure not the + * checkpointed structure. + */ + if(!msr_tm_active(cpumsr) && msr_tm_active(current->thread.regs->msr)) + return; __giveup_altivec(current); } } @@ -278,7 +309,8 @@ EXPORT_SYMBOL_GPL(flush_altivec_to_thread); static int restore_altivec(struct task_struct *tsk) { - if (cpu_has_feature(CPU_FTR_ALTIVEC) && tsk->thread.load_vec) { + if (cpu_has_feature(CPU_FTR_ALTIVEC) && + (tsk->thread.load_vec || msr_tm_active(tsk->thread.regs->msr))) { load_vr_state(&tsk->thread.vr_state); tsk->thread.used_vr = 1; tsk->thread.load_vec++; @@ -321,12 +353,23 @@ static void save_vsx(struct task_struct *tsk) void enable_kernel_vsx(void) { + unsigned long cpumsr; + WARN_ON(preemptible()); - msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX); + cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX); if (current->thread.regs && (current->thread.regs->msr & MSR_VSX)) { check_if_tm_restore_required(current); + /* + * If a thread has already been reclaimed then the + * checkpointed registers are on the CPU but have definitely + * been saved by the reclaim code. Don't need to and *cannot* + * giveup as this would save to the 'live' structure not the + * checkpointed structure. + */ + if(!msr_tm_active(cpumsr) && msr_tm_active(current->thread.regs->msr)) + return; if (current->thread.regs->msr & MSR_FP) __giveup_fpu(current); if (current->thread.regs->msr & MSR_VEC) @@ -438,6 +481,7 @@ void giveup_all(struct task_struct *tsk) return; msr_check_and_set(msr_all_available); + check_if_tm_restore_required(tsk); #ifdef CONFIG_PPC_FPU if (usermsr & MSR_FP) @@ -464,7 +508,8 @@ void restore_math(struct pt_regs *regs) { unsigned long msr; - if (!current->thread.load_fp && !loadvec(current->thread)) + if (!msr_tm_active(regs->msr) && + !current->thread.load_fp && !loadvec(current->thread)) return; msr = regs->msr; @@ -767,29 +812,15 @@ static inline bool hw_brk_match(struct arch_hw_breakpoint *a, } #ifdef CONFIG_PPC_TRANSACTIONAL_MEM + +static inline bool tm_enabled(struct task_struct *tsk) +{ + return tsk && tsk->thread.regs && (tsk->thread.regs->msr & MSR_TM); +} + static void tm_reclaim_thread(struct thread_struct *thr, struct thread_info *ti, uint8_t cause) { - unsigned long msr_diff = 0; - - /* - * If FP/VSX registers have been already saved to the - * thread_struct, move them to the transact_fp array. - * We clear the TIF_RESTORE_TM bit since after the reclaim - * the thread will no longer be transactional. - */ - if (test_ti_thread_flag(ti, TIF_RESTORE_TM)) { - msr_diff = thr->ckpt_regs.msr & ~thr->regs->msr; - if (msr_diff & MSR_FP) - memcpy(&thr->transact_fp, &thr->fp_state, - sizeof(struct thread_fp_state)); - if (msr_diff & MSR_VEC) - memcpy(&thr->transact_vr, &thr->vr_state, - sizeof(struct thread_vr_state)); - clear_ti_thread_flag(ti, TIF_RESTORE_TM); - msr_diff &= MSR_FP | MSR_VEC | MSR_VSX | MSR_FE0 | MSR_FE1; - } - /* * Use the current MSR TM suspended bit to track if we have * checkpointed state outstanding. @@ -808,15 +839,9 @@ static void tm_reclaim_thread(struct thread_struct *thr, if (!MSR_TM_SUSPENDED(mfmsr())) return; - tm_reclaim(thr, thr->regs->msr, cause); + giveup_all(container_of(thr, struct task_struct, thread)); - /* Having done the reclaim, we now have the checkpointed - * FP/VSX values in the registers. These might be valid - * even if we have previously called enable_kernel_fp() or - * flush_fp_to_thread(), so update thr->regs->msr to - * indicate their current validity. - */ - thr->regs->msr |= msr_diff; + tm_reclaim(thr, thr->ckpt_regs.msr, cause); } void tm_reclaim_current(uint8_t cause) @@ -832,8 +857,8 @@ static inline void tm_reclaim_task(struct task_struct *tsk) * * In switching we need to maintain a 2nd register state as * oldtask->thread.ckpt_regs. We tm_reclaim(oldproc); this saves the - * checkpointed (tbegin) state in ckpt_regs and saves the transactional - * (current) FPRs into oldtask->thread.transact_fpr[]. + * checkpointed (tbegin) state in ckpt_regs, ckfp_state and + * ckvr_state * * We also context switch (save) TFHAR/TEXASR/TFIAR in here. */ @@ -845,14 +870,6 @@ static inline void tm_reclaim_task(struct task_struct *tsk) if (!MSR_TM_ACTIVE(thr->regs->msr)) goto out_and_saveregs; - /* Stash the original thread MSR, as giveup_fpu et al will - * modify it. We hold onto it to see whether the task used - * FP & vector regs. If the TIF_RESTORE_TM flag is set, - * ckpt_regs.msr is already set. - */ - if (!test_ti_thread_flag(task_thread_info(tsk), TIF_RESTORE_TM)) - thr->ckpt_regs.msr = thr->regs->msr; - TM_DEBUG("--- tm_reclaim on pid %d (NIP=%lx, " "ccr=%lx, msr=%lx, trap=%lx)\n", tsk->pid, thr->regs->nip, @@ -881,6 +898,9 @@ void tm_recheckpoint(struct thread_struct *thread, { unsigned long flags; + if (!(thread->regs->msr & MSR_TM)) + return; + /* We really can't be interrupted here as the TEXASR registers can't * change and later in the trecheckpoint code, we have a userspace R1. * So let's hard disable over this region. @@ -910,10 +930,10 @@ static inline void tm_recheckpoint_new_task(struct task_struct *new) * If the task was using FP, we non-lazily reload both the original and * the speculative FP register states. This is because the kernel * doesn't see if/when a TM rollback occurs, so if we take an FP - * unavoidable later, we are unable to determine which set of FP regs + * unavailable later, we are unable to determine which set of FP regs * need to be restored. */ - if (!new->thread.regs) + if (!tm_enabled(new)) return; if (!MSR_TM_ACTIVE(new->thread.regs->msr)){ @@ -926,35 +946,35 @@ static inline void tm_recheckpoint_new_task(struct task_struct *new) "(new->msr 0x%lx, new->origmsr 0x%lx)\n", new->pid, new->thread.regs->msr, msr); - /* This loads the checkpointed FP/VEC state, if used */ tm_recheckpoint(&new->thread, msr); - /* This loads the speculative FP/VEC state, if used */ - if (msr & MSR_FP) { - do_load_up_transact_fpu(&new->thread); - new->thread.regs->msr |= - (MSR_FP | new->thread.fpexc_mode); - } -#ifdef CONFIG_ALTIVEC - if (msr & MSR_VEC) { - do_load_up_transact_altivec(&new->thread); - new->thread.regs->msr |= MSR_VEC; - } -#endif - /* We may as well turn on VSX too since all the state is restored now */ - if (msr & MSR_VSX) - new->thread.regs->msr |= MSR_VSX; + /* + * The checkpointed state has been restored but the live state has + * not, ensure all the math functionality is turned off to trigger + * restore_math() to reload. + */ + new->thread.regs->msr &= ~(MSR_FP | MSR_VEC | MSR_VSX); TM_DEBUG("*** tm_recheckpoint of pid %d complete " "(kernel msr 0x%lx)\n", new->pid, mfmsr()); } -static inline void __switch_to_tm(struct task_struct *prev) +static inline void __switch_to_tm(struct task_struct *prev, + struct task_struct *new) { if (cpu_has_feature(CPU_FTR_TM)) { - tm_enable(); - tm_reclaim_task(prev); + if (tm_enabled(prev) || tm_enabled(new)) + tm_enable(); + + if (tm_enabled(prev)) { + prev->thread.load_tm++; + tm_reclaim_task(prev); + if (!MSR_TM_ACTIVE(prev->thread.regs->msr) && prev->thread.load_tm == 0) + prev->thread.regs->msr &= ~MSR_TM; + } + + tm_recheckpoint_new_task(new); } } @@ -976,6 +996,12 @@ void restore_tm_state(struct pt_regs *regs) { unsigned long msr_diff; + /* + * This is the only moment we should clear TIF_RESTORE_TM as + * it is here that ckpt_regs.msr and pt_regs.msr become the same + * again, anything else could lead to an incorrect ckpt_msr being + * saved and therefore incorrect signal contexts. + */ clear_thread_flag(TIF_RESTORE_TM); if (!MSR_TM_ACTIVE(regs->msr)) return; @@ -983,6 +1009,13 @@ void restore_tm_state(struct pt_regs *regs) msr_diff = current->thread.ckpt_regs.msr & ~regs->msr; msr_diff &= MSR_FP | MSR_VEC | MSR_VSX; + /* Ensure that restore_math() will restore */ + if (msr_diff & MSR_FP) + current->thread.load_fp = 1; +#ifdef CONFIG_ALIVEC + if (cpu_has_feature(CPU_FTR_ALTIVEC) && msr_diff & MSR_VEC) + current->thread.load_vec = 1; +#endif restore_math(regs); regs->msr |= msr_diff; @@ -990,7 +1023,7 @@ void restore_tm_state(struct pt_regs *regs) #else #define tm_recheckpoint_new_task(new) -#define __switch_to_tm(prev) +#define __switch_to_tm(prev, new) #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ static inline void save_sprs(struct thread_struct *t) @@ -1131,11 +1164,11 @@ struct task_struct *__switch_to(struct task_struct *prev, */ save_sprs(&prev->thread); - __switch_to_tm(prev); - /* Save FPU, Altivec, VSX and SPE state */ giveup_all(prev); + __switch_to_tm(prev, new); + /* * We can't take a PMU exception inside _switch() since there is a * window where the kernel stack SLB and the kernel stack are out @@ -1143,8 +1176,6 @@ struct task_struct *__switch_to(struct task_struct *prev, */ hard_irq_disable(); - tm_recheckpoint_new_task(new); - /* * Call restore_sprs() before calling _switch(). If we move it after * _switch() then we miss out on calling it for new tasks. The reason @@ -1379,9 +1410,11 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) * transitions the CPU out of TM mode. Hence we need to call * tm_recheckpoint_new_task() (on the same task) to restore the * checkpointed state back and the TM mode. + * + * Can't pass dst because it isn't ready. Doesn't matter, passing + * dst is only important for __switch_to() */ - __switch_to_tm(src); - tm_recheckpoint_new_task(src); + __switch_to_tm(src, src); *dst = *src; @@ -1623,8 +1656,6 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp) current->thread.used_spe = 0; #endif /* CONFIG_SPE */ #ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (cpu_has_feature(CPU_FTR_TM)) - regs->msr |= MSR_TM; current->thread.tm_tfhar = 0; current->thread.tm_texasr = 0; current->thread.tm_tfiar = 0; diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c index d3eff99e938c..88ac964f4858 100644 --- a/arch/powerpc/kernel/prom_init.c +++ b/arch/powerpc/kernel/prom_init.c @@ -42,6 +42,7 @@ #include <asm/sections.h> #include <asm/machdep.h> #include <asm/opal.h> +#include <asm/asm-prototypes.h> #include <linux/linux_logo.h> @@ -2643,6 +2644,86 @@ static void __init fixup_device_tree_efika(void) #define fixup_device_tree_efika() #endif +#ifdef CONFIG_PPC_PASEMI_NEMO +/* + * CFE supplied on Nemo is broken in several ways, biggest + * problem is that it reassigns ISA interrupts to unused mpic ints. + * Add an interrupt-controller property for the io-bridge to use + * and correct the ints so we can attach them to an irq_domain + */ +static void __init fixup_device_tree_pasemi(void) +{ + u32 interrupts[2], parent, rval, val = 0; + char *name, *pci_name; + phandle iob, node; + + /* Find the root pci node */ + name = "/pxp@0,e0000000"; + iob = call_prom("finddevice", 1, 1, ADDR(name)); + if (!PHANDLE_VALID(iob)) + return; + + /* check if interrupt-controller node set yet */ + if (prom_getproplen(iob, "interrupt-controller") !=PROM_ERROR) + return; + + prom_printf("adding interrupt-controller property for SB600...\n"); + + prom_setprop(iob, name, "interrupt-controller", &val, 0); + + pci_name = "/pxp@0,e0000000/pci@11"; + node = call_prom("finddevice", 1, 1, ADDR(pci_name)); + parent = ADDR(iob); + + for( ; prom_next_node(&node); ) { + /* scan each node for one with an interrupt */ + if (!PHANDLE_VALID(node)) + continue; + + rval = prom_getproplen(node, "interrupts"); + if (rval == 0 || rval == PROM_ERROR) + continue; + + prom_getprop(node, "interrupts", &interrupts, sizeof(interrupts)); + if ((interrupts[0] < 212) || (interrupts[0] > 222)) + continue; + + /* found a node, update both interrupts and interrupt-parent */ + if ((interrupts[0] >= 212) && (interrupts[0] <= 215)) + interrupts[0] -= 203; + if ((interrupts[0] >= 216) && (interrupts[0] <= 220)) + interrupts[0] -= 213; + if (interrupts[0] == 221) + interrupts[0] = 14; + if (interrupts[0] == 222) + interrupts[0] = 8; + + prom_setprop(node, pci_name, "interrupts", interrupts, + sizeof(interrupts)); + prom_setprop(node, pci_name, "interrupt-parent", &parent, + sizeof(parent)); + } + + /* + * The io-bridge has device_type set to 'io-bridge' change it to 'isa' + * so that generic isa-bridge code can add the SB600 and its on-board + * peripherals. + */ + name = "/pxp@0,e0000000/io-bridge@0"; + iob = call_prom("finddevice", 1, 1, ADDR(name)); + if (!PHANDLE_VALID(iob)) + return; + + /* device_type is already set, just change it. */ + + prom_printf("Changing device_type of SB600 node...\n"); + + prom_setprop(iob, name, "device_type", "isa", sizeof("isa")); +} +#else /* !CONFIG_PPC_PASEMI_NEMO */ +static inline void fixup_device_tree_pasemi(void) { } +#endif + static void __init fixup_device_tree(void) { fixup_device_tree_maple(); @@ -2650,6 +2731,7 @@ static void __init fixup_device_tree(void) fixup_device_tree_chrp(); fixup_device_tree_pmac(); fixup_device_tree_efika(); + fixup_device_tree_pasemi(); } static void __init prom_find_boot_cpu(void) diff --git a/arch/powerpc/kernel/ptrace.c b/arch/powerpc/kernel/ptrace.c index bf91658a8a40..b1ec62f2cc31 100644 --- a/arch/powerpc/kernel/ptrace.c +++ b/arch/powerpc/kernel/ptrace.c @@ -39,6 +39,7 @@ #include <asm/pgtable.h> #include <asm/switch_to.h> #include <asm/tm.h> +#include <asm/asm-prototypes.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h> @@ -402,13 +403,9 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset, } /* - * When the transaction is active, 'transact_fp' holds the current running - * value of all FPR registers and 'fp_state' holds the last checkpointed - * value of all FPR registers for the current transaction. When transaction - * is not active 'fp_state' holds the current running state of all the FPR - * registers. So this function which returns the current running values of - * all the FPR registers, needs to know whether any transaction is active - * or not. + * Regardless of transactions, 'fp_state' holds the current running + * value of all FPR registers and 'ckfp_state' holds the last checkpointed + * value of all FPR registers for the current transaction. * * Userspace interface buffer layout: * @@ -416,13 +413,6 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset, * u64 fpr[32]; * u64 fpscr; * }; - * - * There are two config options CONFIG_VSX and CONFIG_PPC_TRANSACTIONAL_MEM - * which determines the final code in this function. All the combinations of - * these two config options are possible except the one below as transactional - * memory config pulls in CONFIG_VSX automatically. - * - * !defined(CONFIG_VSX) && defined(CONFIG_PPC_TRANSACTIONAL_MEM) */ static int fpr_get(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, @@ -431,50 +421,29 @@ static int fpr_get(struct task_struct *target, const struct user_regset *regset, #ifdef CONFIG_VSX u64 buf[33]; int i; -#endif - flush_fp_to_thread(target); -#if defined(CONFIG_VSX) && defined(CONFIG_PPC_TRANSACTIONAL_MEM) - /* copy to local buffer then write that out */ - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); - for (i = 0; i < 32 ; i++) - buf[i] = target->thread.TS_TRANS_FPR(i); - buf[32] = target->thread.transact_fp.fpscr; - } else { - for (i = 0; i < 32 ; i++) - buf[i] = target->thread.TS_FPR(i); - buf[32] = target->thread.fp_state.fpscr; - } - return user_regset_copyout(&pos, &count, &kbuf, &ubuf, buf, 0, -1); -#endif + flush_fp_to_thread(target); -#if defined(CONFIG_VSX) && !defined(CONFIG_PPC_TRANSACTIONAL_MEM) /* copy to local buffer then write that out */ for (i = 0; i < 32 ; i++) buf[i] = target->thread.TS_FPR(i); buf[32] = target->thread.fp_state.fpscr; return user_regset_copyout(&pos, &count, &kbuf, &ubuf, buf, 0, -1); -#endif - -#if !defined(CONFIG_VSX) && !defined(CONFIG_PPC_TRANSACTIONAL_MEM) +#else BUILD_BUG_ON(offsetof(struct thread_fp_state, fpscr) != offsetof(struct thread_fp_state, fpr[32])); + flush_fp_to_thread(target); + return user_regset_copyout(&pos, &count, &kbuf, &ubuf, &target->thread.fp_state, 0, -1); #endif } /* - * When the transaction is active, 'transact_fp' holds the current running - * value of all FPR registers and 'fp_state' holds the last checkpointed - * value of all FPR registers for the current transaction. When transaction - * is not active 'fp_state' holds the current running state of all the FPR - * registers. So this function which setss the current running values of - * all the FPR registers, needs to know whether any transaction is active - * or not. + * Regardless of transactions, 'fp_state' holds the current running + * value of all FPR registers and 'ckfp_state' holds the last checkpointed + * value of all FPR registers for the current transaction. * * Userspace interface buffer layout: * @@ -483,12 +452,6 @@ static int fpr_get(struct task_struct *target, const struct user_regset *regset, * u64 fpscr; * }; * - * There are two config options CONFIG_VSX and CONFIG_PPC_TRANSACTIONAL_MEM - * which determines the final code in this function. All the combinations of - * these two config options are possible except the one below as transactional - * memory config pulls in CONFIG_VSX automatically. - * - * !defined(CONFIG_VSX) && defined(CONFIG_PPC_TRANSACTIONAL_MEM) */ static int fpr_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, @@ -497,44 +460,24 @@ static int fpr_set(struct task_struct *target, const struct user_regset *regset, #ifdef CONFIG_VSX u64 buf[33]; int i; -#endif + flush_fp_to_thread(target); -#if defined(CONFIG_VSX) && defined(CONFIG_PPC_TRANSACTIONAL_MEM) /* copy to local buffer then write that out */ i = user_regset_copyin(&pos, &count, &kbuf, &ubuf, buf, 0, -1); if (i) return i; - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); - for (i = 0; i < 32 ; i++) - target->thread.TS_TRANS_FPR(i) = buf[i]; - target->thread.transact_fp.fpscr = buf[32]; - } else { - for (i = 0; i < 32 ; i++) - target->thread.TS_FPR(i) = buf[i]; - target->thread.fp_state.fpscr = buf[32]; - } - return 0; -#endif - -#if defined(CONFIG_VSX) && !defined(CONFIG_PPC_TRANSACTIONAL_MEM) - /* copy to local buffer then write that out */ - i = user_regset_copyin(&pos, &count, &kbuf, &ubuf, buf, 0, -1); - if (i) - return i; for (i = 0; i < 32 ; i++) target->thread.TS_FPR(i) = buf[i]; target->thread.fp_state.fpscr = buf[32]; return 0; -#endif - -#if !defined(CONFIG_VSX) && !defined(CONFIG_PPC_TRANSACTIONAL_MEM) +#else BUILD_BUG_ON(offsetof(struct thread_fp_state, fpscr) != offsetof(struct thread_fp_state, fpr[32])); + flush_fp_to_thread(target); + return user_regset_copyin(&pos, &count, &kbuf, &ubuf, &target->thread.fp_state, 0, -1); #endif @@ -562,13 +505,10 @@ static int vr_active(struct task_struct *target, } /* - * When the transaction is active, 'transact_vr' holds the current running - * value of all the VMX registers and 'vr_state' holds the last checkpointed - * value of all the VMX registers for the current transaction to fall back - * on in case it aborts. When transaction is not active 'vr_state' holds - * the current running state of all the VMX registers. So this function which - * gets the current running values of all the VMX registers, needs to know - * whether any transaction is active or not. + * Regardless of transactions, 'vr_state' holds the current running + * value of all the VMX registers and 'ckvr_state' holds the last + * checkpointed value of all the VMX registers for the current + * transaction to fall back on in case it aborts. * * Userspace interface buffer layout: * @@ -582,7 +522,6 @@ static int vr_get(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, void *kbuf, void __user *ubuf) { - struct thread_vr_state *addr; int ret; flush_altivec_to_thread(target); @@ -590,19 +529,8 @@ static int vr_get(struct task_struct *target, const struct user_regset *regset, BUILD_BUG_ON(offsetof(struct thread_vr_state, vscr) != offsetof(struct thread_vr_state, vr[32])); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - flush_fp_to_thread(target); - flush_tmregs_to_thread(target); - addr = &target->thread.transact_vr; - } else { - addr = &target->thread.vr_state; - } -#else - addr = &target->thread.vr_state; -#endif ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, - addr, 0, + &target->thread.vr_state, 0, 33 * sizeof(vector128)); if (!ret) { /* @@ -614,14 +542,7 @@ static int vr_get(struct task_struct *target, const struct user_regset *regset, } vrsave; memset(&vrsave, 0, sizeof(vrsave)); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) - vrsave.word = target->thread.transact_vrsave; - else - vrsave.word = target->thread.vrsave; -#else vrsave.word = target->thread.vrsave; -#endif ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, &vrsave, 33 * sizeof(vector128), -1); @@ -631,13 +552,10 @@ static int vr_get(struct task_struct *target, const struct user_regset *regset, } /* - * When the transaction is active, 'transact_vr' holds the current running - * value of all the VMX registers and 'vr_state' holds the last checkpointed - * value of all the VMX registers for the current transaction to fall back - * on in case it aborts. When transaction is not active 'vr_state' holds - * the current running state of all the VMX registers. So this function which - * sets the current running values of all the VMX registers, needs to know - * whether any transaction is active or not. + * Regardless of transactions, 'vr_state' holds the current running + * value of all the VMX registers and 'ckvr_state' holds the last + * checkpointed value of all the VMX registers for the current + * transaction to fall back on in case it aborts. * * Userspace interface buffer layout: * @@ -651,7 +569,6 @@ static int vr_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct thread_vr_state *addr; int ret; flush_altivec_to_thread(target); @@ -659,19 +576,8 @@ static int vr_set(struct task_struct *target, const struct user_regset *regset, BUILD_BUG_ON(offsetof(struct thread_vr_state, vscr) != offsetof(struct thread_vr_state, vr[32])); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - flush_fp_to_thread(target); - flush_tmregs_to_thread(target); - addr = &target->thread.transact_vr; - } else { - addr = &target->thread.vr_state; - } -#else - addr = &target->thread.vr_state; -#endif ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, - addr, 0, + &target->thread.vr_state, 0, 33 * sizeof(vector128)); if (!ret && count > 0) { /* @@ -683,27 +589,12 @@ static int vr_set(struct task_struct *target, const struct user_regset *regset, } vrsave; memset(&vrsave, 0, sizeof(vrsave)); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) - vrsave.word = target->thread.transact_vrsave; - else - vrsave.word = target->thread.vrsave; -#else vrsave.word = target->thread.vrsave; -#endif + ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &vrsave, 33 * sizeof(vector128), -1); - if (!ret) { - -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) - target->thread.transact_vrsave = vrsave.word; - else - target->thread.vrsave = vrsave.word; -#else + if (!ret) target->thread.vrsave = vrsave.word; -#endif - } } return ret; @@ -725,13 +616,10 @@ static int vsr_active(struct task_struct *target, } /* - * When the transaction is active, 'transact_fp' holds the current running - * value of all FPR registers and 'fp_state' holds the last checkpointed - * value of all FPR registers for the current transaction. When transaction - * is not active 'fp_state' holds the current running state of all the FPR - * registers. So this function which returns the current running values of - * all the FPR registers, needs to know whether any transaction is active - * or not. + * Regardless of transactions, 'fp_state' holds the current running + * value of all FPR registers and 'ckfp_state' holds the last + * checkpointed value of all FPR registers for the current + * transaction. * * Userspace interface buffer layout: * @@ -746,27 +634,14 @@ static int vsr_get(struct task_struct *target, const struct user_regset *regset, u64 buf[32]; int ret, i; -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); -#endif flush_vsx_to_thread(target); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - for (i = 0; i < 32 ; i++) - buf[i] = target->thread. - transact_fp.fpr[i][TS_VSRLOWOFFSET]; - } else { - for (i = 0; i < 32 ; i++) - buf[i] = target->thread. - fp_state.fpr[i][TS_VSRLOWOFFSET]; - } -#else for (i = 0; i < 32 ; i++) buf[i] = target->thread.fp_state.fpr[i][TS_VSRLOWOFFSET]; -#endif + ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, buf, 0, 32 * sizeof(double)); @@ -774,12 +649,10 @@ static int vsr_get(struct task_struct *target, const struct user_regset *regset, } /* - * When the transaction is active, 'transact_fp' holds the current running - * value of all FPR registers and 'fp_state' holds the last checkpointed - * value of all FPR registers for the current transaction. When transaction - * is not active 'fp_state' holds the current running state of all the FPR - * registers. So this function which sets the current running values of all - * the FPR registers, needs to know whether any transaction is active or not. + * Regardless of transactions, 'fp_state' holds the current running + * value of all FPR registers and 'ckfp_state' holds the last + * checkpointed value of all FPR registers for the current + * transaction. * * Userspace interface buffer layout: * @@ -794,31 +667,16 @@ static int vsr_set(struct task_struct *target, const struct user_regset *regset, u64 buf[32]; int ret,i; -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); -#endif flush_vsx_to_thread(target); ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, buf, 0, 32 * sizeof(double)); - -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(target->thread.regs->msr)) { - for (i = 0; i < 32 ; i++) - target->thread.transact_fp. - fpr[i][TS_VSRLOWOFFSET] = buf[i]; - } else { + if (!ret) for (i = 0; i < 32 ; i++) - target->thread.fp_state. - fpr[i][TS_VSRLOWOFFSET] = buf[i]; - } -#else - for (i = 0; i < 32 ; i++) - target->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = buf[i]; -#endif - + target->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = buf[i]; return ret; } @@ -944,9 +802,9 @@ static int tm_cgpr_get(struct task_struct *target, if (!MSR_TM_ACTIVE(target->thread.regs->msr)) return -ENODATA; + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, &target->thread.ckpt_regs, @@ -1009,9 +867,9 @@ static int tm_cgpr_set(struct task_struct *target, if (!MSR_TM_ACTIVE(target->thread.regs->msr)) return -ENODATA; + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &target->thread.ckpt_regs, @@ -1087,7 +945,7 @@ static int tm_cfpr_active(struct task_struct *target, * * This function gets in transaction checkpointed FPR registers. * - * When the transaction is active 'fp_state' holds the checkpointed + * When the transaction is active 'ckfp_state' holds the checkpointed * values for the current transaction to fall back on if it aborts * in between. This function gets those checkpointed FPR registers. * The userspace interface buffer layout is as follows. @@ -1111,14 +969,14 @@ static int tm_cfpr_get(struct task_struct *target, if (!MSR_TM_ACTIVE(target->thread.regs->msr)) return -ENODATA; + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); /* copy to local buffer then write that out */ for (i = 0; i < 32 ; i++) - buf[i] = target->thread.TS_FPR(i); - buf[32] = target->thread.fp_state.fpscr; + buf[i] = target->thread.TS_CKFPR(i); + buf[32] = target->thread.ckfp_state.fpscr; return user_regset_copyout(&pos, &count, &kbuf, &ubuf, buf, 0, -1); } @@ -1133,7 +991,7 @@ static int tm_cfpr_get(struct task_struct *target, * * This function sets in transaction checkpointed FPR registers. * - * When the transaction is active 'fp_state' holds the checkpointed + * When the transaction is active 'ckfp_state' holds the checkpointed * FPR register values for the current transaction to fall back on * if it aborts in between. This function sets these checkpointed * FPR registers. The userspace interface buffer layout is as follows. @@ -1157,17 +1015,17 @@ static int tm_cfpr_set(struct task_struct *target, if (!MSR_TM_ACTIVE(target->thread.regs->msr)) return -ENODATA; + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); /* copy to local buffer then write that out */ i = user_regset_copyin(&pos, &count, &kbuf, &ubuf, buf, 0, -1); if (i) return i; for (i = 0; i < 32 ; i++) - target->thread.TS_FPR(i) = buf[i]; - target->thread.fp_state.fpscr = buf[32]; + target->thread.TS_CKFPR(i) = buf[i]; + target->thread.ckfp_state.fpscr = buf[32]; return 0; } @@ -1202,7 +1060,7 @@ static int tm_cvmx_active(struct task_struct *target, * * This function gets in transaction checkpointed VMX registers. * - * When the transaction is active 'vr_state' and 'vr_save' hold + * When the transaction is active 'ckvr_state' and 'ckvrsave' hold * the checkpointed values for the current transaction to fall * back on if it aborts in between. The userspace interface buffer * layout is as follows. @@ -1229,12 +1087,12 @@ static int tm_cvmx_get(struct task_struct *target, return -ENODATA; /* Flush the state */ + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, - &target->thread.vr_state, 0, + &target->thread.ckvr_state, 0, 33 * sizeof(vector128)); if (!ret) { /* @@ -1245,7 +1103,7 @@ static int tm_cvmx_get(struct task_struct *target, u32 word; } vrsave; memset(&vrsave, 0, sizeof(vrsave)); - vrsave.word = target->thread.vrsave; + vrsave.word = target->thread.ckvrsave; ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, &vrsave, 33 * sizeof(vector128), -1); } @@ -1264,7 +1122,7 @@ static int tm_cvmx_get(struct task_struct *target, * * This function sets in transaction checkpointed VMX registers. * - * When the transaction is active 'vr_state' and 'vr_save' hold + * When the transaction is active 'ckvr_state' and 'ckvrsave' hold * the checkpointed values for the current transaction to fall * back on if it aborts in between. The userspace interface buffer * layout is as follows. @@ -1290,12 +1148,12 @@ static int tm_cvmx_set(struct task_struct *target, if (!MSR_TM_ACTIVE(target->thread.regs->msr)) return -ENODATA; + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, - &target->thread.vr_state, 0, + &target->thread.ckvr_state, 0, 33 * sizeof(vector128)); if (!ret && count > 0) { /* @@ -1306,11 +1164,11 @@ static int tm_cvmx_set(struct task_struct *target, u32 word; } vrsave; memset(&vrsave, 0, sizeof(vrsave)); - vrsave.word = target->thread.vrsave; + vrsave.word = target->thread.ckvrsave; ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &vrsave, 33 * sizeof(vector128), -1); if (!ret) - target->thread.vrsave = vrsave.word; + target->thread.ckvrsave = vrsave.word; } return ret; @@ -1348,7 +1206,7 @@ static int tm_cvsx_active(struct task_struct *target, * * This function gets in transaction checkpointed VSX registers. * - * When the transaction is active 'fp_state' holds the checkpointed + * When the transaction is active 'ckfp_state' holds the checkpointed * values for the current transaction to fall back on if it aborts * in between. This function gets those checkpointed VSX registers. * The userspace interface buffer layout is as follows. @@ -1372,13 +1230,13 @@ static int tm_cvsx_get(struct task_struct *target, return -ENODATA; /* Flush the state */ + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); flush_vsx_to_thread(target); for (i = 0; i < 32 ; i++) - buf[i] = target->thread.fp_state.fpr[i][TS_VSRLOWOFFSET]; + buf[i] = target->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET]; ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, buf, 0, 32 * sizeof(double)); @@ -1396,7 +1254,7 @@ static int tm_cvsx_get(struct task_struct *target, * * This function sets in transaction checkpointed VSX registers. * - * When the transaction is active 'fp_state' holds the checkpointed + * When the transaction is active 'ckfp_state' holds the checkpointed * VSX register values for the current transaction to fall back on * if it aborts in between. This function sets these checkpointed * FPR registers. The userspace interface buffer layout is as follows. @@ -1420,15 +1278,16 @@ static int tm_cvsx_set(struct task_struct *target, return -ENODATA; /* Flush the state */ + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); flush_vsx_to_thread(target); ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, buf, 0, 32 * sizeof(double)); - for (i = 0; i < 32 ; i++) - target->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = buf[i]; + if (!ret) + for (i = 0; i < 32 ; i++) + target->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET] = buf[i]; return ret; } @@ -1484,9 +1343,9 @@ static int tm_spr_get(struct task_struct *target, return -ENODEV; /* Flush the states */ + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); /* TFHAR register */ ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, @@ -1540,9 +1399,9 @@ static int tm_spr_set(struct task_struct *target, return -ENODEV; /* Flush the states */ + flush_tmregs_to_thread(target); flush_fp_to_thread(target); flush_altivec_to_thread(target); - flush_tmregs_to_thread(target); /* TFHAR register */ ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, @@ -2065,33 +1924,12 @@ static const struct user_regset_view user_ppc_native_view = { static int gpr32_get_common(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, - void *kbuf, void __user *ubuf, bool tm_active) + void *kbuf, void __user *ubuf, + unsigned long *regs) { - const unsigned long *regs = &target->thread.regs->gpr[0]; - const unsigned long *ckpt_regs; compat_ulong_t *k = kbuf; compat_ulong_t __user *u = ubuf; compat_ulong_t reg; - int i; - -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - ckpt_regs = &target->thread.ckpt_regs.gpr[0]; -#endif - if (tm_active) { - regs = ckpt_regs; - } else { - if (target->thread.regs == NULL) - return -EIO; - - if (!FULL_REGS(target->thread.regs)) { - /* - * We have a partial register set. - * Fill 14-31 with bogus values. - */ - for (i = 14; i < 32; i++) - target->thread.regs->gpr[i] = NV_REG_POISON; - } - } pos /= sizeof(reg); count /= sizeof(reg); @@ -2133,29 +1971,13 @@ static int gpr32_get_common(struct task_struct *target, static int gpr32_set_common(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, - const void *kbuf, const void __user *ubuf, bool tm_active) + const void *kbuf, const void __user *ubuf, + unsigned long *regs) { - unsigned long *regs = &target->thread.regs->gpr[0]; - unsigned long *ckpt_regs; const compat_ulong_t *k = kbuf; const compat_ulong_t __user *u = ubuf; compat_ulong_t reg; -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - ckpt_regs = &target->thread.ckpt_regs.gpr[0]; -#endif - - if (tm_active) { - regs = ckpt_regs; - } else { - regs = &target->thread.regs->gpr[0]; - - if (target->thread.regs == NULL) - return -EIO; - - CHECK_FULL_REGS(target->thread.regs); - } - pos /= sizeof(reg); count /= sizeof(reg); @@ -2220,7 +2042,8 @@ static int tm_cgpr32_get(struct task_struct *target, unsigned int pos, unsigned int count, void *kbuf, void __user *ubuf) { - return gpr32_get_common(target, regset, pos, count, kbuf, ubuf, 1); + return gpr32_get_common(target, regset, pos, count, kbuf, ubuf, + &target->thread.ckpt_regs.gpr[0]); } static int tm_cgpr32_set(struct task_struct *target, @@ -2228,7 +2051,8 @@ static int tm_cgpr32_set(struct task_struct *target, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - return gpr32_set_common(target, regset, pos, count, kbuf, ubuf, 1); + return gpr32_set_common(target, regset, pos, count, kbuf, ubuf, + &target->thread.ckpt_regs.gpr[0]); } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ @@ -2237,7 +2061,21 @@ static int gpr32_get(struct task_struct *target, unsigned int pos, unsigned int count, void *kbuf, void __user *ubuf) { - return gpr32_get_common(target, regset, pos, count, kbuf, ubuf, 0); + int i; + + if (target->thread.regs == NULL) + return -EIO; + + if (!FULL_REGS(target->thread.regs)) { + /* + * We have a partial register set. + * Fill 14-31 with bogus values. + */ + for (i = 14; i < 32; i++) + target->thread.regs->gpr[i] = NV_REG_POISON; + } + return gpr32_get_common(target, regset, pos, count, kbuf, ubuf, + &target->thread.regs->gpr[0]); } static int gpr32_set(struct task_struct *target, @@ -2245,7 +2083,12 @@ static int gpr32_set(struct task_struct *target, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - return gpr32_set_common(target, regset, pos, count, kbuf, ubuf, 0); + if (target->thread.regs == NULL) + return -EIO; + + CHECK_FULL_REGS(target->thread.regs); + return gpr32_set_common(target, regset, pos, count, kbuf, ubuf, + &target->thread.regs->gpr[0]); } /* diff --git a/arch/powerpc/kernel/signal.c b/arch/powerpc/kernel/signal.c index cb64d6feb45a..bbe77aed198d 100644 --- a/arch/powerpc/kernel/signal.c +++ b/arch/powerpc/kernel/signal.c @@ -99,22 +99,24 @@ static void check_syscall_restart(struct pt_regs *regs, struct k_sigaction *ka, } } -static void do_signal(struct pt_regs *regs) +static void do_signal(struct task_struct *tsk) { sigset_t *oldset = sigmask_to_save(); struct ksignal ksig; int ret; int is32 = is_32bit_task(); + BUG_ON(tsk != current); + get_signal(&ksig); /* Is there any syscall restart business here ? */ - check_syscall_restart(regs, &ksig.ka, ksig.sig > 0); + check_syscall_restart(tsk->thread.regs, &ksig.ka, ksig.sig > 0); if (ksig.sig <= 0) { /* No signal to deliver -- put the saved sigmask back */ restore_saved_sigmask(); - regs->trap = 0; + tsk->thread.regs->trap = 0; return; /* no signals delivered */ } @@ -124,23 +126,22 @@ static void do_signal(struct pt_regs *regs) * user space. The DABR will have been cleared if it * triggered inside the kernel. */ - if (current->thread.hw_brk.address && - current->thread.hw_brk.type) - __set_breakpoint(¤t->thread.hw_brk); + if (tsk->thread.hw_brk.address && tsk->thread.hw_brk.type) + __set_breakpoint(&tsk->thread.hw_brk); #endif /* Re-enable the breakpoints for the signal stack */ - thread_change_pc(current, regs); + thread_change_pc(tsk, tsk->thread.regs); if (is32) { if (ksig.ka.sa.sa_flags & SA_SIGINFO) - ret = handle_rt_signal32(&ksig, oldset, regs); + ret = handle_rt_signal32(&ksig, oldset, tsk); else - ret = handle_signal32(&ksig, oldset, regs); + ret = handle_signal32(&ksig, oldset, tsk); } else { - ret = handle_rt_signal64(&ksig, oldset, regs); + ret = handle_rt_signal64(&ksig, oldset, tsk); } - regs->trap = 0; + tsk->thread.regs->trap = 0; signal_setup_done(ret, &ksig, test_thread_flag(TIF_SINGLESTEP)); } @@ -151,8 +152,10 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) if (thread_info_flags & _TIF_UPROBE) uprobe_notify_resume(regs); - if (thread_info_flags & _TIF_SIGPENDING) - do_signal(regs); + if (thread_info_flags & _TIF_SIGPENDING) { + BUG_ON(regs != current->thread.regs); + do_signal(current); + } if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); @@ -162,7 +165,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) user_enter(); } -unsigned long get_tm_stackpointer(struct pt_regs *regs) +unsigned long get_tm_stackpointer(struct task_struct *tsk) { /* When in an active transaction that takes a signal, we need to be * careful with the stack. It's possible that the stack has moved back @@ -187,11 +190,13 @@ unsigned long get_tm_stackpointer(struct pt_regs *regs) */ #ifdef CONFIG_PPC_TRANSACTIONAL_MEM - if (MSR_TM_ACTIVE(regs->msr)) { + BUG_ON(tsk != current); + + if (MSR_TM_ACTIVE(tsk->thread.regs->msr)) { tm_reclaim_current(TM_CAUSE_SIGNAL); - if (MSR_TM_TRANSACTIONAL(regs->msr)) - return current->thread.ckpt_regs.gpr[1]; + if (MSR_TM_TRANSACTIONAL(tsk->thread.regs->msr)) + return tsk->thread.ckpt_regs.gpr[1]; } #endif - return regs->gpr[1]; + return tsk->thread.regs->gpr[1]; } diff --git a/arch/powerpc/kernel/signal.h b/arch/powerpc/kernel/signal.h index be305c858e51..7c59d88b9d86 100644 --- a/arch/powerpc/kernel/signal.h +++ b/arch/powerpc/kernel/signal.h @@ -16,39 +16,41 @@ extern void __user *get_sigframe(struct ksignal *ksig, unsigned long sp, size_t frame_size, int is_32); extern int handle_signal32(struct ksignal *ksig, sigset_t *oldset, - struct pt_regs *regs); + struct task_struct *tsk); extern int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, - struct pt_regs *regs); + struct task_struct *tsk); extern unsigned long copy_fpr_to_user(void __user *to, struct task_struct *task); -extern unsigned long copy_transact_fpr_to_user(void __user *to, +extern unsigned long copy_ckfpr_to_user(void __user *to, struct task_struct *task); extern unsigned long copy_fpr_from_user(struct task_struct *task, void __user *from); -extern unsigned long copy_transact_fpr_from_user(struct task_struct *task, +extern unsigned long copy_ckfpr_from_user(struct task_struct *task, void __user *from); +extern unsigned long get_tm_stackpointer(struct task_struct *tsk); + #ifdef CONFIG_VSX extern unsigned long copy_vsx_to_user(void __user *to, struct task_struct *task); -extern unsigned long copy_transact_vsx_to_user(void __user *to, +extern unsigned long copy_ckvsx_to_user(void __user *to, struct task_struct *task); extern unsigned long copy_vsx_from_user(struct task_struct *task, void __user *from); -extern unsigned long copy_transact_vsx_from_user(struct task_struct *task, +extern unsigned long copy_ckvsx_from_user(struct task_struct *task, void __user *from); #endif #ifdef CONFIG_PPC64 extern int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, - struct pt_regs *regs); + struct task_struct *tsk); #else /* CONFIG_PPC64 */ static inline int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, - struct pt_regs *regs) + struct task_struct *tsk) { return -EFAULT; } diff --git a/arch/powerpc/kernel/signal_32.c b/arch/powerpc/kernel/signal_32.c index a7daf749b97f..27aa913ac91d 100644 --- a/arch/powerpc/kernel/signal_32.c +++ b/arch/powerpc/kernel/signal_32.c @@ -44,6 +44,7 @@ #include <asm/vdso.h> #include <asm/switch_to.h> #include <asm/tm.h> +#include <asm/asm-prototypes.h> #ifdef CONFIG_PPC64 #include "ppc32.h" #include <asm/unistd.h> @@ -315,7 +316,7 @@ unsigned long copy_vsx_from_user(struct task_struct *task, } #ifdef CONFIG_PPC_TRANSACTIONAL_MEM -unsigned long copy_transact_fpr_to_user(void __user *to, +unsigned long copy_ckfpr_to_user(void __user *to, struct task_struct *task) { u64 buf[ELF_NFPREG]; @@ -323,12 +324,12 @@ unsigned long copy_transact_fpr_to_user(void __user *to, /* save FPR copy to local buffer then write to the thread_struct */ for (i = 0; i < (ELF_NFPREG - 1) ; i++) - buf[i] = task->thread.TS_TRANS_FPR(i); - buf[i] = task->thread.transact_fp.fpscr; + buf[i] = task->thread.TS_CKFPR(i); + buf[i] = task->thread.ckfp_state.fpscr; return __copy_to_user(to, buf, ELF_NFPREG * sizeof(double)); } -unsigned long copy_transact_fpr_from_user(struct task_struct *task, +unsigned long copy_ckfpr_from_user(struct task_struct *task, void __user *from) { u64 buf[ELF_NFPREG]; @@ -337,13 +338,13 @@ unsigned long copy_transact_fpr_from_user(struct task_struct *task, if (__copy_from_user(buf, from, ELF_NFPREG * sizeof(double))) return 1; for (i = 0; i < (ELF_NFPREG - 1) ; i++) - task->thread.TS_TRANS_FPR(i) = buf[i]; - task->thread.transact_fp.fpscr = buf[i]; + task->thread.TS_CKFPR(i) = buf[i]; + task->thread.ckfp_state.fpscr = buf[i]; return 0; } -unsigned long copy_transact_vsx_to_user(void __user *to, +unsigned long copy_ckvsx_to_user(void __user *to, struct task_struct *task) { u64 buf[ELF_NVSRHALFREG]; @@ -351,11 +352,11 @@ unsigned long copy_transact_vsx_to_user(void __user *to, /* save FPR copy to local buffer then write to the thread_struct */ for (i = 0; i < ELF_NVSRHALFREG; i++) - buf[i] = task->thread.transact_fp.fpr[i][TS_VSRLOWOFFSET]; + buf[i] = task->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET]; return __copy_to_user(to, buf, ELF_NVSRHALFREG * sizeof(double)); } -unsigned long copy_transact_vsx_from_user(struct task_struct *task, +unsigned long copy_ckvsx_from_user(struct task_struct *task, void __user *from) { u64 buf[ELF_NVSRHALFREG]; @@ -364,7 +365,7 @@ unsigned long copy_transact_vsx_from_user(struct task_struct *task, if (__copy_from_user(buf, from, ELF_NVSRHALFREG * sizeof(double))) return 1; for (i = 0; i < ELF_NVSRHALFREG ; i++) - task->thread.transact_fp.fpr[i][TS_VSRLOWOFFSET] = buf[i]; + task->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET] = buf[i]; return 0; } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ @@ -384,17 +385,17 @@ inline unsigned long copy_fpr_from_user(struct task_struct *task, } #ifdef CONFIG_PPC_TRANSACTIONAL_MEM -inline unsigned long copy_transact_fpr_to_user(void __user *to, +inline unsigned long copy_ckfpr_to_user(void __user *to, struct task_struct *task) { - return __copy_to_user(to, task->thread.transact_fp.fpr, + return __copy_to_user(to, task->thread.ckfp_state.fpr, ELF_NFPREG * sizeof(double)); } -inline unsigned long copy_transact_fpr_from_user(struct task_struct *task, +inline unsigned long copy_ckfpr_from_user(struct task_struct *task, void __user *from) { - return __copy_from_user(task->thread.transact_fp.fpr, from, + return __copy_from_user(task->thread.ckfp_state.fpr, from, ELF_NFPREG * sizeof(double)); } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ @@ -525,9 +526,6 @@ static int save_tm_user_regs(struct pt_regs *regs, */ regs->msr &= ~MSR_TS_MASK; - /* Make sure floating point registers are stored in regs */ - flush_fp_to_thread(current); - /* Save both sets of general registers */ if (save_general_regs(¤t->thread.ckpt_regs, frame) || save_general_regs(regs, tm_frame)) @@ -545,18 +543,17 @@ static int save_tm_user_regs(struct pt_regs *regs, #ifdef CONFIG_ALTIVEC /* save altivec registers */ if (current->thread.used_vr) { - flush_altivec_to_thread(current); - if (__copy_to_user(&frame->mc_vregs, ¤t->thread.vr_state, + if (__copy_to_user(&frame->mc_vregs, ¤t->thread.ckvr_state, ELF_NVRREG * sizeof(vector128))) return 1; if (msr & MSR_VEC) { if (__copy_to_user(&tm_frame->mc_vregs, - ¤t->thread.transact_vr, + ¤t->thread.vr_state, ELF_NVRREG * sizeof(vector128))) return 1; } else { if (__copy_to_user(&tm_frame->mc_vregs, - ¤t->thread.vr_state, + ¤t->thread.ckvr_state, ELF_NVRREG * sizeof(vector128))) return 1; } @@ -573,28 +570,28 @@ static int save_tm_user_regs(struct pt_regs *regs, * most significant bits of that same vector. --BenH */ if (cpu_has_feature(CPU_FTR_ALTIVEC)) - current->thread.vrsave = mfspr(SPRN_VRSAVE); - if (__put_user(current->thread.vrsave, + current->thread.ckvrsave = mfspr(SPRN_VRSAVE); + if (__put_user(current->thread.ckvrsave, (u32 __user *)&frame->mc_vregs[32])) return 1; if (msr & MSR_VEC) { - if (__put_user(current->thread.transact_vrsave, + if (__put_user(current->thread.vrsave, (u32 __user *)&tm_frame->mc_vregs[32])) return 1; } else { - if (__put_user(current->thread.vrsave, + if (__put_user(current->thread.ckvrsave, (u32 __user *)&tm_frame->mc_vregs[32])) return 1; } #endif /* CONFIG_ALTIVEC */ - if (copy_fpr_to_user(&frame->mc_fregs, current)) + if (copy_ckfpr_to_user(&frame->mc_fregs, current)) return 1; if (msr & MSR_FP) { - if (copy_transact_fpr_to_user(&tm_frame->mc_fregs, current)) + if (copy_fpr_to_user(&tm_frame->mc_fregs, current)) return 1; } else { - if (copy_fpr_to_user(&tm_frame->mc_fregs, current)) + if (copy_ckfpr_to_user(&tm_frame->mc_fregs, current)) return 1; } @@ -606,15 +603,14 @@ static int save_tm_user_regs(struct pt_regs *regs, * contains valid data */ if (current->thread.used_vsr) { - flush_vsx_to_thread(current); - if (copy_vsx_to_user(&frame->mc_vsregs, current)) + if (copy_ckvsx_to_user(&frame->mc_vsregs, current)) return 1; if (msr & MSR_VSX) { - if (copy_transact_vsx_to_user(&tm_frame->mc_vsregs, + if (copy_vsx_to_user(&tm_frame->mc_vsregs, current)) return 1; } else { - if (copy_vsx_to_user(&tm_frame->mc_vsregs, current)) + if (copy_ckvsx_to_user(&tm_frame->mc_vsregs, current)) return 1; } @@ -698,6 +694,7 @@ static long restore_user_regs(struct pt_regs *regs, if (__copy_from_user(¤t->thread.vr_state, &sr->mc_vregs, sizeof(sr->mc_vregs))) return 1; + current->thread.used_vr = true; } else if (current->thread.used_vr) memset(¤t->thread.vr_state, 0, ELF_NVRREG * sizeof(vector128)); @@ -724,6 +721,7 @@ static long restore_user_regs(struct pt_regs *regs, */ if (copy_vsx_from_user(current, &sr->mc_vsregs)) return 1; + current->thread.used_vsr = true; } else if (current->thread.used_vsr) for (i = 0; i < 32 ; i++) current->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; @@ -743,6 +741,7 @@ static long restore_user_regs(struct pt_regs *regs, if (__copy_from_user(current->thread.evr, &sr->mc_vregs, ELF_NEVRREG * sizeof(u32))) return 1; + current->thread.used_spe = true; } else if (current->thread.used_spe) memset(current->thread.evr, 0, ELF_NEVRREG * sizeof(u32)); @@ -793,33 +792,34 @@ static long restore_tm_user_regs(struct pt_regs *regs, regs->msr &= ~MSR_VEC; if (msr & MSR_VEC) { /* restore altivec registers from the stack */ - if (__copy_from_user(¤t->thread.vr_state, &sr->mc_vregs, + if (__copy_from_user(¤t->thread.ckvr_state, &sr->mc_vregs, sizeof(sr->mc_vregs)) || - __copy_from_user(¤t->thread.transact_vr, + __copy_from_user(¤t->thread.vr_state, &tm_sr->mc_vregs, sizeof(sr->mc_vregs))) return 1; + current->thread.used_vr = true; } else if (current->thread.used_vr) { memset(¤t->thread.vr_state, 0, ELF_NVRREG * sizeof(vector128)); - memset(¤t->thread.transact_vr, 0, + memset(¤t->thread.ckvr_state, 0, ELF_NVRREG * sizeof(vector128)); } /* Always get VRSAVE back */ - if (__get_user(current->thread.vrsave, + if (__get_user(current->thread.ckvrsave, (u32 __user *)&sr->mc_vregs[32]) || - __get_user(current->thread.transact_vrsave, + __get_user(current->thread.vrsave, (u32 __user *)&tm_sr->mc_vregs[32])) return 1; if (cpu_has_feature(CPU_FTR_ALTIVEC)) - mtspr(SPRN_VRSAVE, current->thread.vrsave); + mtspr(SPRN_VRSAVE, current->thread.ckvrsave); #endif /* CONFIG_ALTIVEC */ regs->msr &= ~(MSR_FP | MSR_FE0 | MSR_FE1); if (copy_fpr_from_user(current, &sr->mc_fregs) || - copy_transact_fpr_from_user(current, &tm_sr->mc_fregs)) + copy_ckfpr_from_user(current, &tm_sr->mc_fregs)) return 1; #ifdef CONFIG_VSX @@ -829,13 +829,14 @@ static long restore_tm_user_regs(struct pt_regs *regs, * Restore altivec registers from the stack to a local * buffer, then write this out to the thread_struct */ - if (copy_vsx_from_user(current, &sr->mc_vsregs) || - copy_transact_vsx_from_user(current, &tm_sr->mc_vsregs)) + if (copy_vsx_from_user(current, &tm_sr->mc_vsregs) || + copy_ckvsx_from_user(current, &sr->mc_vsregs)) return 1; + current->thread.used_vsr = true; } else if (current->thread.used_vsr) for (i = 0; i < 32 ; i++) { current->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; - current->thread.transact_fp.fpr[i][TS_VSRLOWOFFSET] = 0; + current->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET] = 0; } #endif /* CONFIG_VSX */ @@ -848,6 +849,7 @@ static long restore_tm_user_regs(struct pt_regs *regs, if (__copy_from_user(current->thread.evr, &sr->mc_vregs, ELF_NEVRREG * sizeof(u32))) return 1; + current->thread.used_spe = true; } else if (current->thread.used_spe) memset(current->thread.evr, 0, ELF_NEVRREG * sizeof(u32)); @@ -877,13 +879,14 @@ static long restore_tm_user_regs(struct pt_regs *regs, tm_recheckpoint(¤t->thread, msr); /* This loads the speculative FP/VEC state, if used */ + msr_check_and_set(msr & (MSR_FP | MSR_VEC)); if (msr & MSR_FP) { - do_load_up_transact_fpu(¤t->thread); + load_fp_state(¤t->thread.fp_state); regs->msr |= (MSR_FP | current->thread.fpexc_mode); } #ifdef CONFIG_ALTIVEC if (msr & MSR_VEC) { - do_load_up_transact_altivec(¤t->thread); + load_vr_state(¤t->thread.vr_state); regs->msr |= MSR_VEC; } #endif @@ -971,7 +974,7 @@ int copy_siginfo_from_user32(siginfo_t *to, struct compat_siginfo __user *from) * (one which gets siginfo). */ int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, - struct pt_regs *regs) + struct task_struct *tsk) { struct rt_sigframe __user *rt_sf; struct mcontext __user *frame; @@ -980,10 +983,13 @@ int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, unsigned long newsp = 0; int sigret; unsigned long tramp; + struct pt_regs *regs = tsk->thread.regs; + + BUG_ON(tsk != current); /* Set up Signal Frame */ /* Put a Real Time Context onto stack */ - rt_sf = get_sigframe(ksig, get_tm_stackpointer(regs), sizeof(*rt_sf), 1); + rt_sf = get_sigframe(ksig, get_tm_stackpointer(tsk), sizeof(*rt_sf), 1); addr = rt_sf; if (unlikely(rt_sf == NULL)) goto badframe; @@ -1000,9 +1006,9 @@ int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, /* Save user registers on the stack */ frame = &rt_sf->uc.uc_mcontext; addr = frame; - if (vdso32_rt_sigtramp && current->mm->context.vdso_base) { + if (vdso32_rt_sigtramp && tsk->mm->context.vdso_base) { sigret = 0; - tramp = current->mm->context.vdso_base + vdso32_rt_sigtramp; + tramp = tsk->mm->context.vdso_base + vdso32_rt_sigtramp; } else { sigret = __NR_rt_sigreturn; tramp = (unsigned long) frame->tramp; @@ -1029,7 +1035,7 @@ int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, } regs->link = tramp; - current->thread.fp_state.fpscr = 0; /* turn off all fp exceptions */ + tsk->thread.fp_state.fpscr = 0; /* turn off all fp exceptions */ /* create a stack frame for the caller of the handler */ newsp = ((unsigned long)rt_sf) - (__SIGNAL_FRAMESIZE + 16); @@ -1054,7 +1060,7 @@ badframe: printk_ratelimited(KERN_INFO "%s[%d]: bad frame in handle_rt_signal32: " "%p nip %08lx lr %08lx\n", - current->comm, current->pid, + tsk->comm, tsk->pid, addr, regs->nip, regs->link); return 1; @@ -1410,7 +1416,8 @@ int sys_debug_setcontext(struct ucontext __user *ctx, /* * OK, we're invoking a handler */ -int handle_signal32(struct ksignal *ksig, sigset_t *oldset, struct pt_regs *regs) +int handle_signal32(struct ksignal *ksig, sigset_t *oldset, + struct task_struct *tsk) { struct sigcontext __user *sc; struct sigframe __user *frame; @@ -1418,9 +1425,12 @@ int handle_signal32(struct ksignal *ksig, sigset_t *oldset, struct pt_regs *regs unsigned long newsp = 0; int sigret; unsigned long tramp; + struct pt_regs *regs = tsk->thread.regs; + + BUG_ON(tsk != current); /* Set up Signal Frame */ - frame = get_sigframe(ksig, get_tm_stackpointer(regs), sizeof(*frame), 1); + frame = get_sigframe(ksig, get_tm_stackpointer(tsk), sizeof(*frame), 1); if (unlikely(frame == NULL)) goto badframe; sc = (struct sigcontext __user *) &frame->sctx; @@ -1439,9 +1449,9 @@ int handle_signal32(struct ksignal *ksig, sigset_t *oldset, struct pt_regs *regs || __put_user(ksig->sig, &sc->signal)) goto badframe; - if (vdso32_sigtramp && current->mm->context.vdso_base) { + if (vdso32_sigtramp && tsk->mm->context.vdso_base) { sigret = 0; - tramp = current->mm->context.vdso_base + vdso32_sigtramp; + tramp = tsk->mm->context.vdso_base + vdso32_sigtramp; } else { sigret = __NR_sigreturn; tramp = (unsigned long) frame->mctx.tramp; @@ -1463,7 +1473,7 @@ int handle_signal32(struct ksignal *ksig, sigset_t *oldset, struct pt_regs *regs regs->link = tramp; - current->thread.fp_state.fpscr = 0; /* turn off all fp exceptions */ + tsk->thread.fp_state.fpscr = 0; /* turn off all fp exceptions */ /* create a stack frame for the caller of the handler */ newsp = ((unsigned long)frame) - __SIGNAL_FRAMESIZE; @@ -1483,7 +1493,7 @@ badframe: printk_ratelimited(KERN_INFO "%s[%d]: bad frame in handle_signal32: " "%p nip %08lx lr %08lx\n", - current->comm, current->pid, + tsk->comm, tsk->pid, frame, regs->nip, regs->link); return 1; diff --git a/arch/powerpc/kernel/signal_64.c b/arch/powerpc/kernel/signal_64.c index 70409bb90a95..96698fdf93b4 100644 --- a/arch/powerpc/kernel/signal_64.c +++ b/arch/powerpc/kernel/signal_64.c @@ -35,6 +35,7 @@ #include <asm/vdso.h> #include <asm/switch_to.h> #include <asm/tm.h> +#include <asm/asm-prototypes.h> #include "signal.h" @@ -90,9 +91,9 @@ static elf_vrreg_t __user *sigcontext_vmx_regs(struct sigcontext __user *sc) * Set up the sigcontext for the signal frame. */ -static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, - int signr, sigset_t *set, unsigned long handler, - int ctx_has_vsx_region) +static long setup_sigcontext(struct sigcontext __user *sc, + struct task_struct *tsk, int signr, sigset_t *set, + unsigned long handler, int ctx_has_vsx_region) { /* When CONFIG_ALTIVEC is set, we _always_ setup v_regs even if the * process never used altivec yet (MSR_VEC is zero in pt_regs of @@ -106,17 +107,20 @@ static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, elf_vrreg_t __user *v_regs = sigcontext_vmx_regs(sc); unsigned long vrsave; #endif + struct pt_regs *regs = tsk->thread.regs; unsigned long msr = regs->msr; long err = 0; + BUG_ON(tsk != current); + #ifdef CONFIG_ALTIVEC err |= __put_user(v_regs, &sc->v_regs); /* save altivec registers */ - if (current->thread.used_vr) { - flush_altivec_to_thread(current); + if (tsk->thread.used_vr) { + flush_altivec_to_thread(tsk); /* Copy 33 vec registers (vr0..31 and vscr) to the stack */ - err |= __copy_to_user(v_regs, ¤t->thread.vr_state, + err |= __copy_to_user(v_regs, &tsk->thread.vr_state, 33 * sizeof(vector128)); /* set MSR_VEC in the MSR value in the frame to indicate that sc->v_reg) * contains valid data. @@ -129,16 +133,16 @@ static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, vrsave = 0; if (cpu_has_feature(CPU_FTR_ALTIVEC)) { vrsave = mfspr(SPRN_VRSAVE); - current->thread.vrsave = vrsave; + tsk->thread.vrsave = vrsave; } err |= __put_user(vrsave, (u32 __user *)&v_regs[33]); #else /* CONFIG_ALTIVEC */ err |= __put_user(0, &sc->v_regs); #endif /* CONFIG_ALTIVEC */ - flush_fp_to_thread(current); + flush_fp_to_thread(tsk); /* copy fpr regs and fpscr */ - err |= copy_fpr_to_user(&sc->fp_regs, current); + err |= copy_fpr_to_user(&sc->fp_regs, tsk); /* * Clear the MSR VSX bit to indicate there is no valid state attached @@ -151,10 +155,10 @@ static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, * then out to userspace. Update v_regs to point after the * VMX data. */ - if (current->thread.used_vsr && ctx_has_vsx_region) { - flush_vsx_to_thread(current); + if (tsk->thread.used_vsr && ctx_has_vsx_region) { + flush_vsx_to_thread(tsk); v_regs += ELF_NVRREG; - err |= copy_vsx_to_user(v_regs, current); + err |= copy_vsx_to_user(v_regs, tsk); /* set MSR_VSX in the MSR value in the frame to * indicate that sc->vs_reg) contains valid data. */ @@ -187,7 +191,7 @@ static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, */ static long setup_tm_sigcontexts(struct sigcontext __user *sc, struct sigcontext __user *tm_sc, - struct pt_regs *regs, + struct task_struct *tsk, int signr, sigset_t *set, unsigned long handler) { /* When CONFIG_ALTIVEC is set, we _always_ setup v_regs even if the @@ -202,9 +206,12 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, elf_vrreg_t __user *v_regs = sigcontext_vmx_regs(sc); elf_vrreg_t __user *tm_v_regs = sigcontext_vmx_regs(tm_sc); #endif - unsigned long msr = regs->msr; + struct pt_regs *regs = tsk->thread.regs; + unsigned long msr = tsk->thread.ckpt_regs.msr; long err = 0; + BUG_ON(tsk != current); + BUG_ON(!MSR_TM_ACTIVE(regs->msr)); /* Remove TM bits from thread's MSR. The MSR in the sigcontext @@ -214,28 +221,25 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, */ regs->msr &= ~MSR_TS_MASK; - flush_fp_to_thread(current); - #ifdef CONFIG_ALTIVEC err |= __put_user(v_regs, &sc->v_regs); err |= __put_user(tm_v_regs, &tm_sc->v_regs); /* save altivec registers */ - if (current->thread.used_vr) { - flush_altivec_to_thread(current); + if (tsk->thread.used_vr) { /* Copy 33 vec registers (vr0..31 and vscr) to the stack */ - err |= __copy_to_user(v_regs, ¤t->thread.vr_state, + err |= __copy_to_user(v_regs, &tsk->thread.ckvr_state, 33 * sizeof(vector128)); /* If VEC was enabled there are transactional VRs valid too, * else they're a copy of the checkpointed VRs. */ if (msr & MSR_VEC) err |= __copy_to_user(tm_v_regs, - ¤t->thread.transact_vr, + &tsk->thread.vr_state, 33 * sizeof(vector128)); else err |= __copy_to_user(tm_v_regs, - ¤t->thread.vr_state, + &tsk->thread.ckvr_state, 33 * sizeof(vector128)); /* set MSR_VEC in the MSR value in the frame to indicate @@ -247,13 +251,13 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, * use altivec. */ if (cpu_has_feature(CPU_FTR_ALTIVEC)) - current->thread.vrsave = mfspr(SPRN_VRSAVE); - err |= __put_user(current->thread.vrsave, (u32 __user *)&v_regs[33]); + tsk->thread.ckvrsave = mfspr(SPRN_VRSAVE); + err |= __put_user(tsk->thread.ckvrsave, (u32 __user *)&v_regs[33]); if (msr & MSR_VEC) - err |= __put_user(current->thread.transact_vrsave, + err |= __put_user(tsk->thread.vrsave, (u32 __user *)&tm_v_regs[33]); else - err |= __put_user(current->thread.vrsave, + err |= __put_user(tsk->thread.ckvrsave, (u32 __user *)&tm_v_regs[33]); #else /* CONFIG_ALTIVEC */ @@ -262,11 +266,11 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, #endif /* CONFIG_ALTIVEC */ /* copy fpr regs and fpscr */ - err |= copy_fpr_to_user(&sc->fp_regs, current); + err |= copy_ckfpr_to_user(&sc->fp_regs, tsk); if (msr & MSR_FP) - err |= copy_transact_fpr_to_user(&tm_sc->fp_regs, current); + err |= copy_fpr_to_user(&tm_sc->fp_regs, tsk); else - err |= copy_fpr_to_user(&tm_sc->fp_regs, current); + err |= copy_ckfpr_to_user(&tm_sc->fp_regs, tsk); #ifdef CONFIG_VSX /* @@ -274,17 +278,16 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, * then out to userspace. Update v_regs to point after the * VMX data. */ - if (current->thread.used_vsr) { - flush_vsx_to_thread(current); + if (tsk->thread.used_vsr) { v_regs += ELF_NVRREG; tm_v_regs += ELF_NVRREG; - err |= copy_vsx_to_user(v_regs, current); + err |= copy_ckvsx_to_user(v_regs, tsk); if (msr & MSR_VSX) - err |= copy_transact_vsx_to_user(tm_v_regs, current); + err |= copy_vsx_to_user(tm_v_regs, tsk); else - err |= copy_vsx_to_user(tm_v_regs, current); + err |= copy_ckvsx_to_user(tm_v_regs, tsk); /* set MSR_VSX in the MSR value in the frame to * indicate that sc->vs_reg) contains valid data. @@ -298,7 +301,7 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, WARN_ON(!FULL_REGS(regs)); err |= __copy_to_user(&tm_sc->gp_regs, regs, GP_REGS_SIZE); err |= __copy_to_user(&sc->gp_regs, - ¤t->thread.ckpt_regs, GP_REGS_SIZE); + &tsk->thread.ckpt_regs, GP_REGS_SIZE); err |= __put_user(msr, &tm_sc->gp_regs[PT_MSR]); err |= __put_user(msr, &sc->gp_regs[PT_MSR]); err |= __put_user(signr, &sc->signal); @@ -314,7 +317,7 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc, * Restore the sigcontext from the signal frame. */ -static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, +static long restore_sigcontext(struct task_struct *tsk, sigset_t *set, int sig, struct sigcontext __user *sc) { #ifdef CONFIG_ALTIVEC @@ -323,10 +326,13 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, unsigned long err = 0; unsigned long save_r13 = 0; unsigned long msr; + struct pt_regs *regs = tsk->thread.regs; #ifdef CONFIG_VSX int i; #endif + BUG_ON(tsk != current); + /* If this is not a signal return, we preserve the TLS in r13 */ if (!sig) save_r13 = regs->gpr[13]; @@ -356,7 +362,7 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, /* * Force reload of FP/VEC. - * This has to be done before copying stuff into current->thread.fpr/vr + * This has to be done before copying stuff into tsk->thread.fpr/vr * for the reasons explained in the previous comment. */ regs->msr &= ~(MSR_FP | MSR_FE0 | MSR_FE1 | MSR_VEC | MSR_VSX); @@ -368,21 +374,23 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, if (v_regs && !access_ok(VERIFY_READ, v_regs, 34 * sizeof(vector128))) return -EFAULT; /* Copy 33 vec registers (vr0..31 and vscr) from the stack */ - if (v_regs != NULL && (msr & MSR_VEC) != 0) - err |= __copy_from_user(¤t->thread.vr_state, v_regs, + if (v_regs != NULL && (msr & MSR_VEC) != 0) { + err |= __copy_from_user(&tsk->thread.vr_state, v_regs, 33 * sizeof(vector128)); - else if (current->thread.used_vr) - memset(¤t->thread.vr_state, 0, 33 * sizeof(vector128)); + tsk->thread.used_vr = true; + } else if (tsk->thread.used_vr) { + memset(&tsk->thread.vr_state, 0, 33 * sizeof(vector128)); + } /* Always get VRSAVE back */ if (v_regs != NULL) - err |= __get_user(current->thread.vrsave, (u32 __user *)&v_regs[33]); + err |= __get_user(tsk->thread.vrsave, (u32 __user *)&v_regs[33]); else - current->thread.vrsave = 0; + tsk->thread.vrsave = 0; if (cpu_has_feature(CPU_FTR_ALTIVEC)) - mtspr(SPRN_VRSAVE, current->thread.vrsave); + mtspr(SPRN_VRSAVE, tsk->thread.vrsave); #endif /* CONFIG_ALTIVEC */ /* restore floating point */ - err |= copy_fpr_from_user(current, &sc->fp_regs); + err |= copy_fpr_from_user(tsk, &sc->fp_regs); #ifdef CONFIG_VSX /* * Get additional VSX data. Update v_regs to point after the @@ -390,11 +398,13 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, * buffer for formatting, then into the taskstruct. */ v_regs += ELF_NVRREG; - if ((msr & MSR_VSX) != 0) - err |= copy_vsx_from_user(current, v_regs); - else + if ((msr & MSR_VSX) != 0) { + err |= copy_vsx_from_user(tsk, v_regs); + tsk->thread.used_vsr = true; + } else { for (i = 0; i < 32 ; i++) - current->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; + tsk->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; + } #endif return err; } @@ -404,7 +414,7 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, * Restore the two sigcontexts from the frame of a transactional processes. */ -static long restore_tm_sigcontexts(struct pt_regs *regs, +static long restore_tm_sigcontexts(struct task_struct *tsk, struct sigcontext __user *sc, struct sigcontext __user *tm_sc) { @@ -413,12 +423,16 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, #endif unsigned long err = 0; unsigned long msr; + struct pt_regs *regs = tsk->thread.regs; #ifdef CONFIG_VSX int i; #endif + + BUG_ON(tsk != current); + /* copy the GPRs */ err |= __copy_from_user(regs->gpr, tm_sc->gp_regs, sizeof(regs->gpr)); - err |= __copy_from_user(¤t->thread.ckpt_regs, sc->gp_regs, + err |= __copy_from_user(&tsk->thread.ckpt_regs, sc->gp_regs, sizeof(regs->gpr)); /* @@ -430,7 +444,7 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, * we don't need to re-copy them here. */ err |= __get_user(regs->nip, &tm_sc->gp_regs[PT_NIP]); - err |= __get_user(current->thread.tm_tfhar, &sc->gp_regs[PT_NIP]); + err |= __get_user(tsk->thread.tm_tfhar, &sc->gp_regs[PT_NIP]); /* get MSR separately, transfer the LE bit if doing signal return */ err |= __get_user(msr, &sc->gp_regs[PT_MSR]); @@ -449,13 +463,13 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, err |= __get_user(regs->link, &tm_sc->gp_regs[PT_LNK]); err |= __get_user(regs->xer, &tm_sc->gp_regs[PT_XER]); err |= __get_user(regs->ccr, &tm_sc->gp_regs[PT_CCR]); - err |= __get_user(current->thread.ckpt_regs.ctr, + err |= __get_user(tsk->thread.ckpt_regs.ctr, &sc->gp_regs[PT_CTR]); - err |= __get_user(current->thread.ckpt_regs.link, + err |= __get_user(tsk->thread.ckpt_regs.link, &sc->gp_regs[PT_LNK]); - err |= __get_user(current->thread.ckpt_regs.xer, + err |= __get_user(tsk->thread.ckpt_regs.xer, &sc->gp_regs[PT_XER]); - err |= __get_user(current->thread.ckpt_regs.ccr, + err |= __get_user(tsk->thread.ckpt_regs.ccr, &sc->gp_regs[PT_CCR]); /* These regs are not checkpointed; they can go in 'regs'. */ @@ -466,7 +480,7 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, /* * Force reload of FP/VEC. - * This has to be done before copying stuff into current->thread.fpr/vr + * This has to be done before copying stuff into tsk->thread.fpr/vr * for the reasons explained in the previous comment. */ regs->msr &= ~(MSR_FP | MSR_FE0 | MSR_FE1 | MSR_VEC | MSR_VSX); @@ -483,32 +497,33 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, return -EFAULT; /* Copy 33 vec registers (vr0..31 and vscr) from the stack */ if (v_regs != NULL && tm_v_regs != NULL && (msr & MSR_VEC) != 0) { - err |= __copy_from_user(¤t->thread.vr_state, v_regs, + err |= __copy_from_user(&tsk->thread.ckvr_state, v_regs, 33 * sizeof(vector128)); - err |= __copy_from_user(¤t->thread.transact_vr, tm_v_regs, + err |= __copy_from_user(&tsk->thread.vr_state, tm_v_regs, 33 * sizeof(vector128)); + current->thread.used_vr = true; } - else if (current->thread.used_vr) { - memset(¤t->thread.vr_state, 0, 33 * sizeof(vector128)); - memset(¤t->thread.transact_vr, 0, 33 * sizeof(vector128)); + else if (tsk->thread.used_vr) { + memset(&tsk->thread.vr_state, 0, 33 * sizeof(vector128)); + memset(&tsk->thread.ckvr_state, 0, 33 * sizeof(vector128)); } /* Always get VRSAVE back */ if (v_regs != NULL && tm_v_regs != NULL) { - err |= __get_user(current->thread.vrsave, + err |= __get_user(tsk->thread.ckvrsave, (u32 __user *)&v_regs[33]); - err |= __get_user(current->thread.transact_vrsave, + err |= __get_user(tsk->thread.vrsave, (u32 __user *)&tm_v_regs[33]); } else { - current->thread.vrsave = 0; - current->thread.transact_vrsave = 0; + tsk->thread.vrsave = 0; + tsk->thread.ckvrsave = 0; } if (cpu_has_feature(CPU_FTR_ALTIVEC)) - mtspr(SPRN_VRSAVE, current->thread.vrsave); + mtspr(SPRN_VRSAVE, tsk->thread.vrsave); #endif /* CONFIG_ALTIVEC */ /* restore floating point */ - err |= copy_fpr_from_user(current, &sc->fp_regs); - err |= copy_transact_fpr_from_user(current, &tm_sc->fp_regs); + err |= copy_fpr_from_user(tsk, &tm_sc->fp_regs); + err |= copy_ckfpr_from_user(tsk, &sc->fp_regs); #ifdef CONFIG_VSX /* * Get additional VSX data. Update v_regs to point after the @@ -518,32 +533,31 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, if (v_regs && ((msr & MSR_VSX) != 0)) { v_regs += ELF_NVRREG; tm_v_regs += ELF_NVRREG; - err |= copy_vsx_from_user(current, v_regs); - err |= copy_transact_vsx_from_user(current, tm_v_regs); + err |= copy_vsx_from_user(tsk, tm_v_regs); + err |= copy_ckvsx_from_user(tsk, v_regs); + tsk->thread.used_vsr = true; } else { for (i = 0; i < 32 ; i++) { - current->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; - current->thread.transact_fp.fpr[i][TS_VSRLOWOFFSET] = 0; + tsk->thread.fp_state.fpr[i][TS_VSRLOWOFFSET] = 0; + tsk->thread.ckfp_state.fpr[i][TS_VSRLOWOFFSET] = 0; } } #endif tm_enable(); /* Make sure the transaction is marked as failed */ - current->thread.tm_texasr |= TEXASR_FS; + tsk->thread.tm_texasr |= TEXASR_FS; /* This loads the checkpointed FP/VEC state, if used */ - tm_recheckpoint(¤t->thread, msr); + tm_recheckpoint(&tsk->thread, msr); - /* This loads the speculative FP/VEC state, if used */ + msr_check_and_set(msr & (MSR_FP | MSR_VEC)); if (msr & MSR_FP) { - do_load_up_transact_fpu(¤t->thread); - regs->msr |= (MSR_FP | current->thread.fpexc_mode); + load_fp_state(&tsk->thread.fp_state); + regs->msr |= (MSR_FP | tsk->thread.fpexc_mode); } -#ifdef CONFIG_ALTIVEC if (msr & MSR_VEC) { - do_load_up_transact_altivec(¤t->thread); + load_vr_state(&tsk->thread.vr_state); regs->msr |= MSR_VEC; } -#endif return err; } @@ -594,6 +608,8 @@ int sys_swapcontext(struct ucontext __user *old_ctx, unsigned long new_msr = 0; int ctx_has_vsx_region = 0; + BUG_ON(regs != current->thread.regs); + if (new_ctx && get_user(new_msr, &new_ctx->uc_mcontext.gp_regs[PT_MSR])) return -EFAULT; @@ -616,7 +632,7 @@ int sys_swapcontext(struct ucontext __user *old_ctx, if (old_ctx != NULL) { if (!access_ok(VERIFY_WRITE, old_ctx, ctx_size) - || setup_sigcontext(&old_ctx->uc_mcontext, regs, 0, NULL, 0, + || setup_sigcontext(&old_ctx->uc_mcontext, current, 0, NULL, 0, ctx_has_vsx_region) || __copy_to_user(&old_ctx->uc_sigmask, ¤t->blocked, sizeof(sigset_t))) @@ -644,7 +660,7 @@ int sys_swapcontext(struct ucontext __user *old_ctx, if (__copy_from_user(&set, &new_ctx->uc_sigmask, sizeof(set))) do_exit(SIGSEGV); set_current_blocked(&set); - if (restore_sigcontext(regs, NULL, 0, &new_ctx->uc_mcontext)) + if (restore_sigcontext(current, NULL, 0, &new_ctx->uc_mcontext)) do_exit(SIGSEGV); /* This returns like rt_sigreturn */ @@ -667,6 +683,8 @@ int sys_rt_sigreturn(unsigned long r3, unsigned long r4, unsigned long r5, unsigned long msr; #endif + BUG_ON(current->thread.regs != regs); + /* Always make any pending restarted system calls return -EINTR */ current->restart_block.fn = do_no_restart_syscall; @@ -698,14 +716,14 @@ int sys_rt_sigreturn(unsigned long r3, unsigned long r4, unsigned long r5, struct ucontext __user *uc_transact; if (__get_user(uc_transact, &uc->uc_link)) goto badframe; - if (restore_tm_sigcontexts(regs, &uc->uc_mcontext, + if (restore_tm_sigcontexts(current, &uc->uc_mcontext, &uc_transact->uc_mcontext)) goto badframe; } else /* Fall through, for non-TM restore */ #endif - if (restore_sigcontext(regs, NULL, 1, &uc->uc_mcontext)) + if (restore_sigcontext(current, NULL, 1, &uc->uc_mcontext)) goto badframe; if (restore_altstack(&uc->uc_stack)) @@ -724,13 +742,17 @@ badframe: return 0; } -int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) +int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, + struct task_struct *tsk) { struct rt_sigframe __user *frame; unsigned long newsp = 0; long err = 0; + struct pt_regs *regs = tsk->thread.regs; + + BUG_ON(tsk != current); - frame = get_sigframe(ksig, get_tm_stackpointer(regs), sizeof(*frame), 0); + frame = get_sigframe(ksig, get_tm_stackpointer(tsk), sizeof(*frame), 0); if (unlikely(frame == NULL)) goto badframe; @@ -751,14 +773,13 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs err |= __put_user(&frame->uc_transact, &frame->uc.uc_link); err |= setup_tm_sigcontexts(&frame->uc.uc_mcontext, &frame->uc_transact.uc_mcontext, - regs, ksig->sig, - NULL, + tsk, ksig->sig, NULL, (unsigned long)ksig->ka.sa.sa_handler); } else #endif { err |= __put_user(0, &frame->uc.uc_link); - err |= setup_sigcontext(&frame->uc.uc_mcontext, regs, ksig->sig, + err |= setup_sigcontext(&frame->uc.uc_mcontext, tsk, ksig->sig, NULL, (unsigned long)ksig->ka.sa.sa_handler, 1); } @@ -767,11 +788,11 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs goto badframe; /* Make sure signal handler doesn't get spurious FP exceptions */ - current->thread.fp_state.fpscr = 0; + tsk->thread.fp_state.fpscr = 0; /* Set up to return from userspace. */ - if (vdso64_rt_sigtramp && current->mm->context.vdso_base) { - regs->link = current->mm->context.vdso_base + vdso64_rt_sigtramp; + if (vdso64_rt_sigtramp && tsk->mm->context.vdso_base) { + regs->link = tsk->mm->context.vdso_base + vdso64_rt_sigtramp; } else { err |= setup_trampoline(__NR_rt_sigreturn, &frame->tramp[0]); if (err) @@ -821,7 +842,7 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs badframe: if (show_unhandled_signals) printk_ratelimited(regs->msr & MSR_64BIT ? fmt64 : fmt32, - current->comm, current->pid, "setup_rt_frame", + tsk->comm, tsk->pid, "setup_rt_frame", (long)frame, regs->nip, regs->link); return 1; diff --git a/arch/powerpc/kernel/syscalls.c b/arch/powerpc/kernel/syscalls.c index 5fa92706444b..644cce3d8dce 100644 --- a/arch/powerpc/kernel/syscalls.c +++ b/arch/powerpc/kernel/syscalls.c @@ -40,6 +40,7 @@ #include <asm/syscalls.h> #include <asm/time.h> #include <asm/unistd.h> +#include <asm/asm-prototypes.h> static inline unsigned long do_mmap2(unsigned long addr, size_t len, unsigned long prot, unsigned long flags, diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index 3efbedefba6a..67859b7d1c97 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -73,6 +73,7 @@ #include <asm/vdso_datapage.h> #include <asm/firmware.h> #include <asm/cputime.h> +#include <asm/asm-prototypes.h> /* powerpc clocksource/clockevent code */ diff --git a/arch/powerpc/kernel/tm.S b/arch/powerpc/kernel/tm.S index 298afcf3bf2a..3a2d04134da9 100644 --- a/arch/powerpc/kernel/tm.S +++ b/arch/powerpc/kernel/tm.S @@ -108,6 +108,7 @@ _GLOBAL(tm_reclaim) /* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD]. */ std r3, STK_PARAM(R3)(r1) + std r4, STK_PARAM(R4)(r1) SAVE_NVGPRS(r1) /* We need to setup MSR for VSX register save instructions. */ @@ -126,43 +127,6 @@ _GLOBAL(tm_reclaim) mtmsrd r15 std r14, TM_FRAME_L0(r1) - /* Stash the stack pointer away for use after reclaim */ - std r1, PACAR1(r13) - - /* ******************** FPR/VR/VSRs ************ - * Before reclaiming, capture the current/transactional FPR/VR - * versions /if used/. - * - * (If VSX used, FP and VMX are implied. Or, we don't need to look - * at MSR.VSX as copying FP regs if .FP, vector regs if .VMX covers it.) - * - * We're passed the thread's MSR as parameter 2. - * - * We enabled VEC/FP/VSX in the msr above, so we can execute these - * instructions! - */ - andis. r0, r4, MSR_VEC@h - beq dont_backup_vec - - addi r7, r3, THREAD_TRANSACT_VRSTATE - SAVE_32VRS(0, r6, r7) /* r6 scratch, r7 transact vr state */ - mfvscr v0 - li r6, VRSTATE_VSCR - stvx v0, r7, r6 -dont_backup_vec: - mfspr r0, SPRN_VRSAVE - std r0, THREAD_TRANSACT_VRSAVE(r3) - - andi. r0, r4, MSR_FP - beq dont_backup_fp - - addi r7, r3, THREAD_TRANSACT_FPSTATE - SAVE_32FPRS_VSRS(0, R6, R7) /* r6 scratch, r7 transact fp state */ - - mffs fr0 - stfd fr0,FPSTATE_FPSCR(r7) - -dont_backup_fp: /* Do sanity check on MSR to make sure we are suspended */ li r7, (MSR_TS_S)@higher srdi r6, r14, 32 @@ -170,6 +134,9 @@ dont_backup_fp: 1: tdeqi r6, 0 EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0 + /* Stash the stack pointer away for use after reclaim */ + std r1, PACAR1(r13) + /* Clear MSR RI since we are about to change r1, EE is already off. */ li r4, 0 mtmsrd r4, 1 @@ -273,6 +240,43 @@ dont_backup_fp: * MSR. */ + + /* ******************** FPR/VR/VSRs ************ + * After reclaiming, capture the checkpointed FPRs/VRs /if used/. + * + * (If VSX used, FP and VMX are implied. Or, we don't need to look + * at MSR.VSX as copying FP regs if .FP, vector regs if .VMX covers it.) + * + * We're passed the thread's MSR as the second parameter + * + * We enabled VEC/FP/VSX in the msr above, so we can execute these + * instructions! + */ + ld r4, STK_PARAM(R4)(r1) /* Second parameter, MSR * */ + mr r3, r12 + andis. r0, r4, MSR_VEC@h + beq dont_backup_vec + + addi r7, r3, THREAD_CKVRSTATE + SAVE_32VRS(0, r6, r7) /* r6 scratch, r7 transact vr state */ + mfvscr v0 + li r6, VRSTATE_VSCR + stvx v0, r7, r6 +dont_backup_vec: + mfspr r0, SPRN_VRSAVE + std r0, THREAD_CKVRSAVE(r3) + + andi. r0, r4, MSR_FP + beq dont_backup_fp + + addi r7, r3, THREAD_CKFPSTATE + SAVE_32FPRS_VSRS(0, R6, R7) /* r6 scratch, r7 transact fp state */ + + mffs fr0 + stfd fr0,FPSTATE_FPSCR(r7) + +dont_backup_fp: + /* TM regs, incl TEXASR -- these live in thread_struct. Note they've * been updated by the treclaim, to explain to userland the failure * cause (aborted). @@ -288,6 +292,7 @@ dont_backup_fp: /* Restore original MSR/IRQ state & clear TM mode */ ld r14, TM_FRAME_L0(r1) /* Orig MSR */ + li r15, 0 rldimi r14, r15, MSR_TS_LG, (63-MSR_TS_LG)-1 mtmsrd r14 @@ -356,28 +361,29 @@ _GLOBAL(__tm_recheckpoint) mtmsr r5 #ifdef CONFIG_ALTIVEC - /* FP and VEC registers: These are recheckpointed from thread.fpr[] - * and thread.vr[] respectively. The thread.transact_fpr[] version - * is more modern, and will be loaded subsequently by any FPUnavailable - * trap. + /* + * FP and VEC registers: These are recheckpointed from + * thread.ckfp_state and thread.ckvr_state respectively. The + * thread.fp_state[] version holds the 'live' (transactional) + * and will be loaded subsequently by any FPUnavailable trap. */ andis. r0, r4, MSR_VEC@h beq dont_restore_vec - addi r8, r3, THREAD_VRSTATE + addi r8, r3, THREAD_CKVRSTATE li r5, VRSTATE_VSCR lvx v0, r8, r5 mtvscr v0 REST_32VRS(0, r5, r8) /* r5 scratch, r8 ptr */ dont_restore_vec: - ld r5, THREAD_VRSAVE(r3) + ld r5, THREAD_CKVRSAVE(r3) mtspr SPRN_VRSAVE, r5 #endif andi. r0, r4, MSR_FP beq dont_restore_fp - addi r8, r3, THREAD_FPSTATE + addi r8, r3, THREAD_CKFPSTATE lfd fr0, FPSTATE_FPSCR(r8) MTFSF_L(fr0) REST_32FPRS_VSRS(0, R4, R8) diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 62859ebe0062..a1f8f5641e9e 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -117,7 +117,7 @@ static int die_owner = -1; static unsigned int die_nest_count; static int die_counter; -static unsigned __kprobes long oops_begin(struct pt_regs *regs) +static unsigned long oops_begin(struct pt_regs *regs) { int cpu; unsigned long flags; @@ -144,8 +144,9 @@ static unsigned __kprobes long oops_begin(struct pt_regs *regs) pmac_backlight_unblank(); return flags; } +NOKPROBE_SYMBOL(oops_begin); -static void __kprobes oops_end(unsigned long flags, struct pt_regs *regs, +static void oops_end(unsigned long flags, struct pt_regs *regs, int signr) { bust_spinlocks(0); @@ -196,8 +197,9 @@ static void __kprobes oops_end(unsigned long flags, struct pt_regs *regs, panic("Fatal exception"); do_exit(signr); } +NOKPROBE_SYMBOL(oops_end); -static int __kprobes __die(const char *str, struct pt_regs *regs, long err) +static int __die(const char *str, struct pt_regs *regs, long err) { printk("Oops: %s, sig: %ld [#%d]\n", str, err, ++die_counter); #ifdef CONFIG_PREEMPT @@ -221,6 +223,7 @@ static int __kprobes __die(const char *str, struct pt_regs *regs, long err) return 0; } +NOKPROBE_SYMBOL(__die); void die(const char *str, struct pt_regs *regs, long err) { @@ -802,7 +805,7 @@ void RunModeException(struct pt_regs *regs) _exception(SIGTRAP, regs, 0, 0); } -void __kprobes single_step_exception(struct pt_regs *regs) +void single_step_exception(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); @@ -819,6 +822,7 @@ void __kprobes single_step_exception(struct pt_regs *regs) bail: exception_exit(prev_state); } +NOKPROBE_SYMBOL(single_step_exception); /* * After we have successfully emulated an instruction, we have to @@ -1140,7 +1144,7 @@ static int emulate_math(struct pt_regs *regs) static inline int emulate_math(struct pt_regs *regs) { return -1; } #endif -void __kprobes program_check_exception(struct pt_regs *regs) +void program_check_exception(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); unsigned int reason = get_reason(regs); @@ -1260,16 +1264,18 @@ sigill: bail: exception_exit(prev_state); } +NOKPROBE_SYMBOL(program_check_exception); /* * This occurs when running in hypervisor mode on POWER6 or later * and an illegal instruction is encountered. */ -void __kprobes emulation_assist_interrupt(struct pt_regs *regs) +void emulation_assist_interrupt(struct pt_regs *regs) { regs->msr |= REASON_ILLEGAL; program_check_exception(regs); } +NOKPROBE_SYMBOL(emulation_assist_interrupt); void alignment_exception(struct pt_regs *regs) { @@ -1310,6 +1316,18 @@ bail: exception_exit(prev_state); } +void slb_miss_bad_addr(struct pt_regs *regs) +{ + enum ctx_state prev_state = exception_enter(); + + if (user_mode(regs)) + _exception(SIGSEGV, regs, SEGV_BNDERR, regs->dar); + else + bad_page_fault(regs, regs->dar, SIGSEGV); + + exception_exit(prev_state); +} + void StackOverflow(struct pt_regs *regs) { printk(KERN_CRIT "Kernel stack overflow in process %p, r1=%lx\n", @@ -1372,6 +1390,22 @@ void vsx_unavailable_exception(struct pt_regs *regs) } #ifdef CONFIG_PPC64 +static void tm_unavailable(struct pt_regs *regs) +{ +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + if (user_mode(regs)) { + current->thread.load_tm++; + regs->msr |= MSR_TM; + tm_enable(); + tm_restore_sprs(¤t->thread); + return; + } +#endif + pr_emerg("Unrecoverable TM Unavailable Exception " + "%lx at %lx\n", regs->trap, regs->nip); + die("Unrecoverable TM Unavailable Exception", regs, SIGABRT); +} + void facility_unavailable_exception(struct pt_regs *regs) { static char *facility_strings[] = { @@ -1451,6 +1485,27 @@ void facility_unavailable_exception(struct pt_regs *regs) return; } + if (status == FSCR_TM_LG) { + /* + * If we're here then the hardware is TM aware because it + * generated an exception with FSRM_TM set. + * + * If cpu_has_feature(CPU_FTR_TM) is false, then either firmware + * told us not to do TM, or the kernel is not built with TM + * support. + * + * If both of those things are true, then userspace can spam the + * console by triggering the printk() below just by continually + * doing tbegin (or any TM instruction). So in that case just + * send the process a SIGILL immediately. + */ + if (!cpu_has_feature(CPU_FTR_TM)) + goto out; + + tm_unavailable(regs); + return; + } + if ((status < ARRAY_SIZE(facility_strings)) && facility_strings[status]) facility = facility_strings[status]; @@ -1463,6 +1518,7 @@ void facility_unavailable_exception(struct pt_regs *regs) "%sFacility '%s' unavailable, exception at 0x%lx, MSR=%lx\n", hv ? "Hypervisor " : "", facility, regs->nip, regs->msr); +out: if (user_mode(regs)) { _exception(SIGILL, regs, ILL_ILLOPC, regs->nip); return; @@ -1504,7 +1560,8 @@ void fp_unavailable_tm(struct pt_regs *regs) /* If VMX is in use, get the transactional values back */ if (regs->msr & MSR_VEC) { - do_load_up_transact_altivec(¤t->thread); + msr_check_and_set(MSR_VEC); + load_vr_state(¤t->thread.vr_state); /* At this point all the VSX state is loaded, so enable it */ regs->msr |= MSR_VSX; } @@ -1525,7 +1582,8 @@ void altivec_unavailable_tm(struct pt_regs *regs) current->thread.used_vr = 1; if (regs->msr & MSR_FP) { - do_load_up_transact_fpu(¤t->thread); + msr_check_and_set(MSR_FP); + load_fp_state(¤t->thread.fp_state); regs->msr |= MSR_VSX; } } @@ -1564,10 +1622,12 @@ void vsx_unavailable_tm(struct pt_regs *regs) */ tm_recheckpoint(¤t->thread, regs->msr & ~orig_msr); + msr_check_and_set(orig_msr & (MSR_FP | MSR_VEC)); + if (orig_msr & MSR_FP) - do_load_up_transact_fpu(¤t->thread); + load_fp_state(¤t->thread.fp_state); if (orig_msr & MSR_VEC) - do_load_up_transact_altivec(¤t->thread); + load_vr_state(¤t->thread.vr_state); } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ @@ -1656,7 +1716,7 @@ static void handle_debug(struct pt_regs *regs, unsigned long debug_status) mtspr(SPRN_DBCR0, current->thread.debug.dbcr0); } -void __kprobes DebugException(struct pt_regs *regs, unsigned long debug_status) +void DebugException(struct pt_regs *regs, unsigned long debug_status) { current->thread.debug.dbsr = debug_status; @@ -1717,6 +1777,7 @@ void __kprobes DebugException(struct pt_regs *regs, unsigned long debug_status) } else handle_debug(regs, debug_status); } +NOKPROBE_SYMBOL(DebugException); #endif /* CONFIG_PPC_ADV_DEBUG_REGS */ #if !defined(CONFIG_TAU_INT) diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile index 366ae09b14c1..31107bf5a61f 100644 --- a/arch/powerpc/kernel/vdso64/Makefile +++ b/arch/powerpc/kernel/vdso64/Makefile @@ -31,15 +31,9 @@ $(obj)/%.so: OBJCOPYFLAGS := -S $(obj)/%.so: $(obj)/%.so.dbg FORCE $(call if_changed,objcopy) -# assembly rules for the .S files -$(obj-vdso64): %.o: %.S FORCE - $(call if_changed_dep,vdso64as) - # actual build commands quiet_cmd_vdso64ld = VDSO64L $@ cmd_vdso64ld = $(CC) $(c_flags) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) -quiet_cmd_vdso64as = VDSO64A $@ - cmd_vdso64as = $(CC) $(a_flags) -c -o $@ $< # install commands for the unstripped file quiet_cmd_vdso_install = INSTALL $@ diff --git a/arch/powerpc/kernel/vdso64/datapage.S b/arch/powerpc/kernel/vdso64/datapage.S index 184a6ba7f283..abf17feffe40 100644 --- a/arch/powerpc/kernel/vdso64/datapage.S +++ b/arch/powerpc/kernel/vdso64/datapage.S @@ -59,7 +59,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map) bl V_LOCAL_FUNC(__get_datapage) mtlr r12 addi r3,r3,CFG_SYSCALL_MAP64 - cmpli cr0,r4,0 + cmpldi cr0,r4,0 crclr cr0*4+so beqlr li r0,NR_syscalls diff --git a/arch/powerpc/kernel/vdso64/gettimeofday.S b/arch/powerpc/kernel/vdso64/gettimeofday.S index a76b4af37ef2..382021324883 100644 --- a/arch/powerpc/kernel/vdso64/gettimeofday.S +++ b/arch/powerpc/kernel/vdso64/gettimeofday.S @@ -145,7 +145,7 @@ V_FUNCTION_BEGIN(__kernel_clock_getres) bne cr0,99f li r3,0 - cmpli cr0,r4,0 + cmpldi cr0,r4,0 crclr cr0*4+so beqlr lis r5,CLOCK_REALTIME_RES@h diff --git a/arch/powerpc/kernel/vector.S b/arch/powerpc/kernel/vector.S index 616a6d854638..bc85bdff4e01 100644 --- a/arch/powerpc/kernel/vector.S +++ b/arch/powerpc/kernel/vector.S @@ -7,31 +7,6 @@ #include <asm/page.h> #include <asm/ptrace.h> -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -/* void do_load_up_transact_altivec(struct thread_struct *thread) - * - * This is similar to load_up_altivec but for the transactional version of the - * vector regs. It doesn't mess with the task MSR or valid flags. - * Furthermore, VEC laziness is not supported with TM currently. - */ -_GLOBAL(do_load_up_transact_altivec) - mfmsr r6 - oris r5,r6,MSR_VEC@h - MTMSRD(r5) - isync - - li r4,1 - stw r4,THREAD_USED_VR(r3) - - li r10,THREAD_TRANSACT_VRSTATE+VRSTATE_VSCR - lvx v0,r10,r3 - mtvscr v0 - addi r10,r3,THREAD_TRANSACT_VRSTATE - REST_32VRS(0,r4,r10) - - blr -#endif - /* * Load state from memory into VMX registers including VSCR. * Assumes the caller has enabled VMX in the MSR. diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S index b5fba689fca6..2d1cfafd1404 100644 --- a/arch/powerpc/kernel/vmlinux.lds.S +++ b/arch/powerpc/kernel/vmlinux.lds.S @@ -44,11 +44,58 @@ SECTIONS * Text, read only data and other permanent read-only sections */ - /* Text and gots */ + _text = .; + _stext = .; + + /* + * Head text. + * This needs to be in its own output section to avoid ld placing + * branch trampoline stubs randomly throughout the fixed sections, + * which it will do (even if the branch comes from another section) + * in order to optimize stub generation. + */ + .head.text : AT(ADDR(.head.text) - LOAD_OFFSET) { +#ifdef CONFIG_PPC64 + KEEP(*(.head.text.first_256B)); +#ifdef CONFIG_PPC_BOOK3E +# define END_FIXED 0x100 +#else + KEEP(*(.head.text.real_vectors)); + *(.head.text.real_trampolines); + KEEP(*(.head.text.virt_vectors)); + *(.head.text.virt_trampolines); +# if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) + KEEP(*(.head.data.fwnmi_page)); +# define END_FIXED 0x8000 +# else +# define END_FIXED 0x7000 +# endif +#endif + ASSERT((. == END_FIXED), "vmlinux.lds.S: fixed section overflow error"); +#else /* !CONFIG_PPC64 */ + HEAD_TEXT +#endif + } :kernel + + /* + * If the build dies here, it's likely code in head_64.S is referencing + * labels it can't reach, and the linker inserting stubs without the + * assembler's knowledge. To debug, remove the above assert and + * rebuild. Look for branch stubs in the fixed section region. + * + * Linker stub generation could be allowed in "trampoline" + * sections if absolutely necessary, but this would require + * some rework of the fixed sections. Before resorting to this, + * consider references that have sufficient addressing range, + * (e.g., hand coded trampolines) so the linker does not have + * to add stubs. + * + * Linker stubs at the top of the main text section are currently not + * detected, and will result in a crash at boot due to offsets being + * wrong. + */ .text : AT(ADDR(.text) - LOAD_OFFSET) { ALIGN_FUNCTION(); - HEAD_TEXT - _text = .; /* careful! __ftr_alt_* sections need to be close to .text */ *(.text .fixup __ftr_alt_* .ref.text) SCHED_TEXT @@ -56,6 +103,8 @@ SECTIONS KPROBES_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT + MEM_KEEP(init.text) + MEM_KEEP(exit.text) #ifdef CONFIG_PPC32 *(.got1) diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index ba21be15310f..ad5290005ca4 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -22,7 +22,7 @@ obj64-$(CONFIG_SMP) += locks.o obj64-$(CONFIG_ALTIVEC) += vmx-helper.o ifeq ($(CONFIG_GENERIC_CSUM),) -obj-y += checksum_$(CONFIG_WORD_SIZE).o checksum_wrappers.o +obj-y += checksum_$(BITS).o checksum_wrappers.o endif obj-$(CONFIG_PPC_EMULATE_SSTEP) += sstep.o ldstfp.o diff --git a/arch/powerpc/lib/mem_64.S b/arch/powerpc/lib/mem_64.S index 43435c6892fb..eda7a96161ab 100644 --- a/arch/powerpc/lib/mem_64.S +++ b/arch/powerpc/lib/mem_64.S @@ -37,6 +37,7 @@ _GLOBAL(memset) clrldi r5,r5,58 mtctr r0 beq 5f + .balign 16 4: std r4,0(r6) std r4,8(r6) std r4,16(r6) @@ -90,6 +91,7 @@ _GLOBAL(backwards_memcpy) andi. r0,r6,3 mtctr r7 bne 5f + .balign 16 1: lwz r7,-4(r4) lwzu r8,-8(r4) stw r7,-4(r6) diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile index f2cea6d5e764..1a4e570f7894 100644 --- a/arch/powerpc/mm/Makefile +++ b/arch/powerpc/mm/Makefile @@ -7,17 +7,16 @@ subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror ccflags-$(CONFIG_PPC64) := $(NO_MINIMAL_TOC) obj-y := fault.o mem.o pgtable.o mmap.o \ - init_$(CONFIG_WORD_SIZE).o \ - pgtable_$(CONFIG_WORD_SIZE).o + init_$(BITS).o pgtable_$(BITS).o obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \ tlb_nohash_low.o -obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(CONFIG_WORD_SIZE)e.o +obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o hash64-$(CONFIG_PPC_NATIVE) := hash_native_64.o obj-$(CONFIG_PPC_BOOK3E_64) += pgtable-book3e.o obj-$(CONFIG_PPC_STD_MMU_64) += pgtable-hash64.o hash_utils_64.o slb_low.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o obj-$(CONFIG_PPC_RADIX_MMU) += pgtable-radix.o tlb-radix.o obj-$(CONFIG_PPC_STD_MMU_32) += ppc_mmu_32.o hash_low_32.o mmu_context_hash32.o -obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(CONFIG_WORD_SIZE).o +obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(BITS).o ifeq ($(CONFIG_PPC_STD_MMU_64),y) obj-$(CONFIG_PPC_4K_PAGES) += hash64_4k.o obj-$(CONFIG_PPC_64K_PAGES) += hash64_64k.o diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index bb1ffc559f38..d0b137d96df1 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -205,7 +205,7 @@ static int mm_fault_error(struct pt_regs *regs, unsigned long addr, int fault) * The return value is 0 if the fault was handled, or the signal * number if this is a kernel fault that can't be handled here. */ -int __kprobes do_page_fault(struct pt_regs *regs, unsigned long address, +int do_page_fault(struct pt_regs *regs, unsigned long address, unsigned long error_code) { enum ctx_state prev_state = exception_enter(); @@ -498,8 +498,8 @@ bad_area_nosemaphore: bail: exception_exit(prev_state); return rc; - } +NOKPROBE_SYMBOL(do_page_fault); /* * bad_page_fault is called when we have a bad access from the kernel. diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c index ef3ae891a3db..90480e23fd2c 100644 --- a/arch/powerpc/mm/hash_utils_64.c +++ b/arch/powerpc/mm/hash_utils_64.c @@ -766,6 +766,29 @@ int remove_section_mapping(unsigned long start, unsigned long end) } #endif /* CONFIG_MEMORY_HOTPLUG */ +static void update_hid_for_hash(void) +{ + unsigned long hid0; + unsigned long rb = 3UL << PPC_BITLSHIFT(53); /* IS = 3 */ + + asm volatile("ptesync": : :"memory"); + /* prs = 0, ric = 2, rs = 0, r = 1 is = 3 */ + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(0), "i"(0), "i"(2), "r"(0) : "memory"); + asm volatile("eieio; tlbsync; ptesync; isync; slbia": : :"memory"); + /* + * now switch the HID + */ + hid0 = mfspr(SPRN_HID0); + hid0 &= ~HID0_POWER9_RADIX; + mtspr(SPRN_HID0, hid0); + asm volatile("isync": : :"memory"); + + /* Wait for it to happen */ + while ((mfspr(SPRN_HID0) & HID0_POWER9_RADIX)) + cpu_relax(); +} + static void __init hash_init_partition_table(phys_addr_t hash_table, unsigned long htab_size) { @@ -792,6 +815,8 @@ static void __init hash_init_partition_table(phys_addr_t hash_table, */ partition_tb->patb1 = 0; pr_info("Partition table %p\n", partition_tb); + if (cpu_has_feature(CPU_FTR_POWER9_DD1)) + update_hid_for_hash(); /* * update partition table control register, * 64 K size. @@ -1515,6 +1540,29 @@ out_exit: local_irq_restore(flags); } +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +static inline void tm_flush_hash_page(int local) +{ + /* + * Transactions are not aborted by tlbiel, only tlbie. Without, syncing a + * page back to a block device w/PIO could pick up transactional data + * (bad!) so we force an abort here. Before the sync the page will be + * made read-only, which will flush_hash_page. BIG ISSUE here: if the + * kernel uses a page from userspace without unmapping it first, it may + * see the speculated version. + */ + if (local && cpu_has_feature(CPU_FTR_TM) && current->thread.regs && + MSR_TM_ACTIVE(current->thread.regs->msr)) { + tm_enable(); + tm_abort(TM_CAUSE_TLBI); + } +} +#else +static inline void tm_flush_hash_page(int local) +{ +} +#endif + /* WARNING: This is called from hash_low_64.S, if you change this prototype, * do not forget to update the assembly call site ! */ @@ -1541,21 +1589,7 @@ void flush_hash_page(unsigned long vpn, real_pte_t pte, int psize, int ssize, ssize, local); } pte_iterate_hashed_end(); -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - /* Transactions are not aborted by tlbiel, only tlbie. - * Without, syncing a page back to a block device w/ PIO could pick up - * transactional data (bad!) so we force an abort here. Before the - * sync the page will be made read-only, which will flush_hash_page. - * BIG ISSUE here: if the kernel uses a page from userspace without - * unmapping it first, it may see the speculated version. - */ - if (local && cpu_has_feature(CPU_FTR_TM) && - current->thread.regs && - MSR_TM_ACTIVE(current->thread.regs->msr)) { - tm_enable(); - tm_abort(TM_CAUSE_TLBI); - } -#endif + tm_flush_hash_page(local); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1612,22 +1646,7 @@ void flush_hash_hugepage(unsigned long vsid, unsigned long addr, MMU_PAGE_16M, ssize, local); } tm_abort: -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - /* Transactions are not aborted by tlbiel, only tlbie. - * Without, syncing a page back to a block device w/ PIO could pick up - * transactional data (bad!) so we force an abort here. Before the - * sync the page will be made read-only, which will flush_hash_page. - * BIG ISSUE here: if the kernel uses a page from userspace without - * unmapping it first, it may see the speculated version. - */ - if (local && cpu_has_feature(CPU_FTR_TM) && - current->thread.regs && - MSR_TM_ACTIVE(current->thread.regs->msr)) { - tm_enable(); - tm_abort(TM_CAUSE_TLBI); - } -#endif - return; + tm_flush_hash_page(local); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 7372ee13eb1e..a5d3ecdabc44 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -1019,8 +1019,15 @@ int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, pte = READ_ONCE(*ptep); mask = _PAGE_PRESENT | _PAGE_READ; + + /* + * On some CPUs like the 8xx, _PAGE_RW hence _PAGE_WRITE is defined + * as 0 and _PAGE_RO has to be set when a page is not writable + */ if (write) mask |= _PAGE_WRITE; + else + mask |= _PAGE_RO; if ((pte_val(pte) & mask) != mask) return 0; diff --git a/arch/powerpc/mm/init_32.c b/arch/powerpc/mm/init_32.c index 448685fbf27c..8a7c38b8d335 100644 --- a/arch/powerpc/mm/init_32.c +++ b/arch/powerpc/mm/init_32.c @@ -137,7 +137,7 @@ void __init MMU_init(void) if (memblock.memory.cnt > 1) { #ifndef CONFIG_WII memblock_enforce_memory_limit(memblock.memory.regions[0].size); - printk(KERN_WARNING "Only using first contiguous memory region"); + pr_warn("Only using first contiguous memory region\n"); #else wii_memory_fixups(); #endif diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c index da6a2168ae9e..e0f1c33601dd 100644 --- a/arch/powerpc/mm/mmu_context_iommu.c +++ b/arch/powerpc/mm/mmu_context_iommu.c @@ -15,6 +15,9 @@ #include <linux/rculist.h> #include <linux/vmalloc.h> #include <linux/mutex.h> +#include <linux/migrate.h> +#include <linux/hugetlb.h> +#include <linux/swap.h> #include <asm/mmu_context.h> static DEFINE_MUTEX(mem_list_mutex); @@ -72,6 +75,55 @@ bool mm_iommu_preregistered(void) } EXPORT_SYMBOL_GPL(mm_iommu_preregistered); +/* + * Taken from alloc_migrate_target with changes to remove CMA allocations + */ +struct page *new_iommu_non_cma_page(struct page *page, unsigned long private, + int **resultp) +{ + gfp_t gfp_mask = GFP_USER; + struct page *new_page; + + if (PageHuge(page) || PageTransHuge(page) || PageCompound(page)) + return NULL; + + if (PageHighMem(page)) + gfp_mask |= __GFP_HIGHMEM; + + /* + * We don't want the allocation to force an OOM if possibe + */ + new_page = alloc_page(gfp_mask | __GFP_NORETRY | __GFP_NOWARN); + return new_page; +} + +static int mm_iommu_move_page_from_cma(struct page *page) +{ + int ret = 0; + LIST_HEAD(cma_migrate_pages); + + /* Ignore huge pages for now */ + if (PageHuge(page) || PageTransHuge(page) || PageCompound(page)) + return -EBUSY; + + lru_add_drain(); + ret = isolate_lru_page(page); + if (ret) + return ret; + + list_add(&page->lru, &cma_migrate_pages); + put_page(page); /* Drop the gup reference */ + + ret = migrate_pages(&cma_migrate_pages, new_iommu_non_cma_page, + NULL, 0, MIGRATE_SYNC, MR_CMA); + if (ret) { + if (!list_empty(&cma_migrate_pages)) + putback_movable_pages(&cma_migrate_pages); + } + + return 0; +} + long mm_iommu_get(unsigned long ua, unsigned long entries, struct mm_iommu_table_group_mem_t **pmem) { @@ -124,15 +176,36 @@ long mm_iommu_get(unsigned long ua, unsigned long entries, for (i = 0; i < entries; ++i) { if (1 != get_user_pages_fast(ua + (i << PAGE_SHIFT), 1/* pages */, 1/* iswrite */, &page)) { + ret = -EFAULT; for (j = 0; j < i; ++j) - put_page(pfn_to_page( - mem->hpas[j] >> PAGE_SHIFT)); + put_page(pfn_to_page(mem->hpas[j] >> + PAGE_SHIFT)); vfree(mem->hpas); kfree(mem); - ret = -EFAULT; goto unlock_exit; } - + /* + * If we get a page from the CMA zone, since we are going to + * be pinning these entries, we might as well move them out + * of the CMA zone if possible. NOTE: faulting in + migration + * can be expensive. Batching can be considered later + */ + if (get_pageblock_migratetype(page) == MIGRATE_CMA) { + if (mm_iommu_move_page_from_cma(page)) + goto populate; + if (1 != get_user_pages_fast(ua + (i << PAGE_SHIFT), + 1/* pages */, 1/* iswrite */, + &page)) { + ret = -EFAULT; + for (j = 0; j < i; ++j) + put_page(pfn_to_page(mem->hpas[j] >> + PAGE_SHIFT)); + vfree(mem->hpas); + kfree(mem); + goto unlock_exit; + } + } +populate: mem->hpas[i] = page_to_pfn(page) << PAGE_SHIFT; } diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 34079302cc17..f4f437cbabf1 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -35,7 +35,7 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, #endif changed = !pmd_same(*(pmdp), entry); if (changed) { - __ptep_set_access_flags(pmdp_ptep(pmdp), pmd_pte(entry)); + __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp), pmd_pte(entry)); flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); } return changed; @@ -116,3 +116,12 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, return; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +/* For use by kexec */ +void mmu_cleanup_all(void) +{ + if (radix_enabled()) + radix__mmu_cleanup_all(); + else if (mmu_hash_ops.hpte_clear_all) + mmu_hash_ops.hpte_clear_all(); +} diff --git a/arch/powerpc/mm/pgtable-radix.c b/arch/powerpc/mm/pgtable-radix.c index af897d91d09f..ed7bddc456b7 100644 --- a/arch/powerpc/mm/pgtable-radix.c +++ b/arch/powerpc/mm/pgtable-radix.c @@ -294,6 +294,32 @@ found: return; } +static void update_hid_for_radix(void) +{ + unsigned long hid0; + unsigned long rb = 3UL << PPC_BITLSHIFT(53); /* IS = 3 */ + + asm volatile("ptesync": : :"memory"); + /* prs = 0, ric = 2, rs = 0, r = 1 is = 3 */ + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(1), "i"(0), "i"(2), "r"(0) : "memory"); + /* prs = 1, ric = 2, rs = 0, r = 1 is = 3 */ + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(1), "i"(1), "i"(2), "r"(0) : "memory"); + asm volatile("eieio; tlbsync; ptesync; isync; slbia": : :"memory"); + /* + * now switch the HID + */ + hid0 = mfspr(SPRN_HID0); + hid0 |= HID0_POWER9_RADIX; + mtspr(SPRN_HID0, hid0); + asm volatile("isync": : :"memory"); + + /* Wait for it to happen */ + while (!(mfspr(SPRN_HID0) & HID0_POWER9_RADIX)) + cpu_relax(); +} + void __init radix__early_init_mmu(void) { unsigned long lpcr; @@ -345,6 +371,8 @@ void __init radix__early_init_mmu(void) if (!firmware_has_feature(FW_FEATURE_LPAR)) { radix_init_native(); + if (cpu_has_feature(CPU_FTR_POWER9_DD1)) + update_hid_for_radix(); lpcr = mfspr(SPRN_LPCR); mtspr(SPRN_LPCR, lpcr | LPCR_UPRT | LPCR_HR); radix_init_partition_table(); @@ -368,6 +396,18 @@ void radix__early_init_mmu_secondary(void) } } +void radix__mmu_cleanup_all(void) +{ + unsigned long lpcr; + + if (!firmware_has_feature(FW_FEATURE_LPAR)) { + lpcr = mfspr(SPRN_LPCR); + mtspr(SPRN_LPCR, lpcr & ~LPCR_UPRT); + mtspr(SPRN_PTCR, 0); + radix__flush_tlb_all(); + } +} + void radix__setup_initial_memory_limit(phys_addr_t first_memblock_base, phys_addr_t first_memblock_size) { diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 0b6fb244d0a1..911fdfb63ec1 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -224,7 +224,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, if (changed) { if (!is_vm_hugetlb_page(vma)) assert_pte_locked(vma->vm_mm, address); - __ptep_set_access_flags(ptep, entry); + __ptep_set_access_flags(vma->vm_mm, ptep, entry); flush_tlb_page(vma, address); } return changed; diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S index 9f1983404e1a..e2974fcd20f1 100644 --- a/arch/powerpc/mm/slb_low.S +++ b/arch/powerpc/mm/slb_low.S @@ -178,11 +178,9 @@ BEGIN_FTR_SECTION END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) b slb_finish_load -8: /* invalid EA */ - li r10,0 /* BAD_VSID */ - li r9,0 /* BAD_VSID */ - li r11,SLB_VSID_USER /* flags don't much matter */ - b slb_finish_load +8: /* invalid EA - return an error indication */ + crset 4*cr0+eq /* indicate failure */ + blr /* * Finish loading of an SLB entry and return diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 48df05ef5231..0e49ec541ab5 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -400,3 +400,27 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, radix__flush_tlb_range_psize(vma->vm_mm, start, end, MMU_PAGE_2M); } EXPORT_SYMBOL(radix__flush_pmd_tlb_range); + +void radix__flush_tlb_all(void) +{ + unsigned long rb,prs,r,rs; + unsigned long ric = RIC_FLUSH_ALL; + + rb = 0x3 << PPC_BITLSHIFT(53); /* IS = 3 */ + prs = 0; /* partition scoped */ + r = 1; /* raidx format */ + rs = 1 & ((1UL << 32) - 1); /* any LPID value to flush guest mappings */ + + asm volatile("ptesync": : :"memory"); + /* + * now flush guest entries by passing PRS = 1 and LPID != 0 + */ + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(1), "i"(ric), "r"(rs) : "memory"); + /* + * now flush host entires by passing PRS = 0 and LPID == 0 + */ + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(0) : "memory"); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h index d5301b6f20d0..89f70073dec8 100644 --- a/arch/powerpc/net/bpf_jit.h +++ b/arch/powerpc/net/bpf_jit.h @@ -40,6 +40,8 @@ #define PPC_BLR() EMIT(PPC_INST_BLR) #define PPC_BLRL() EMIT(PPC_INST_BLRL) #define PPC_MTLR(r) EMIT(PPC_INST_MTLR | ___PPC_RT(r)) +#define PPC_BCTR() EMIT(PPC_INST_BCTR) +#define PPC_MTCTR(r) EMIT(PPC_INST_MTCTR | ___PPC_RT(r)) #define PPC_ADDI(d, a, i) EMIT(PPC_INST_ADDI | ___PPC_RT(d) | \ ___PPC_RA(a) | IMM_L(i)) #define PPC_MR(d, a) PPC_OR(d, a, a) diff --git a/arch/powerpc/net/bpf_jit64.h b/arch/powerpc/net/bpf_jit64.h index 5046d6f65c02..62fa7589db2b 100644 --- a/arch/powerpc/net/bpf_jit64.h +++ b/arch/powerpc/net/bpf_jit64.h @@ -16,30 +16,33 @@ /* * Stack layout: + * Ensure the top half (upto local_tmp_var) stays consistent + * with our redzone usage. * * [ prev sp ] <------------- * [ nv gpr save area ] 8*8 | + * [ tail_call_cnt ] 8 | + * [ local_tmp_var ] 8 | * fp (r31) --> [ ebpf stack space ] 512 | - * [ local/tmp var space ] 16 | * [ frame header ] 32/112 | * sp (r1) ---> [ stack pointer ] -------------- */ -/* for bpf JIT code internal usage */ -#define BPF_PPC_STACK_LOCALS 16 /* for gpr non volatile registers BPG_REG_6 to 10, plus skb cache registers */ #define BPF_PPC_STACK_SAVE (8*8) +/* for bpf JIT code internal usage */ +#define BPF_PPC_STACK_LOCALS 16 /* Ensure this is quadword aligned */ -#define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + BPF_PPC_STACK_LOCALS + \ - MAX_BPF_STACK + BPF_PPC_STACK_SAVE) +#define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + MAX_BPF_STACK + \ + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE) #ifndef __ASSEMBLY__ /* BPF register usage */ -#define SKB_HLEN_REG (MAX_BPF_REG + 0) -#define SKB_DATA_REG (MAX_BPF_REG + 1) -#define TMP_REG_1 (MAX_BPF_REG + 2) -#define TMP_REG_2 (MAX_BPF_REG + 3) +#define SKB_HLEN_REG (MAX_BPF_JIT_REG + 0) +#define SKB_DATA_REG (MAX_BPF_JIT_REG + 1) +#define TMP_REG_1 (MAX_BPF_JIT_REG + 2) +#define TMP_REG_2 (MAX_BPF_JIT_REG + 3) /* BPF to ppc register mappings */ static const int b2p[] = { @@ -59,12 +62,16 @@ static const int b2p[] = { /* frame pointer aka BPF_REG_10 */ [BPF_REG_FP] = 31, /* eBPF jit internal registers */ + [BPF_REG_AX] = 2, [SKB_HLEN_REG] = 25, [SKB_DATA_REG] = 26, [TMP_REG_1] = 9, [TMP_REG_2] = 10 }; +/* PPC NVR range -- update this if we ever use NVRs below r24 */ +#define BPF_PPC_NVR_MIN 24 + /* Assembly helpers */ #define DECLARE_LOAD_FUNC(func) u64 func(u64 r3, u64 r4); \ u64 func##_negative_offset(u64 r3, u64 r4); \ @@ -82,6 +89,7 @@ DECLARE_LOAD_FUNC(sk_load_byte); #define SEEN_FUNC 0x1000 /* might call external helpers */ #define SEEN_STACK 0x2000 /* uses BPF stack */ #define SEEN_SKB 0x4000 /* uses sk_buff */ +#define SEEN_TAILCALL 0x8000 /* uses tail calls */ struct codegen_context { /* diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 6073b78516f6..0fe98a567125 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -17,6 +17,7 @@ #include <linux/filter.h> #include <linux/if_vlan.h> #include <asm/kprobes.h> +#include <linux/bpf.h> #include "bpf_jit64.h" @@ -58,6 +59,40 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, BPF_REG_FP); } +/* + * When not setting up our own stackframe, the redzone usage is: + * + * [ prev sp ] <------------- + * [ ... ] | + * sp (r1) ---> [ stack pointer ] -------------- + * [ nv gpr save area ] 8*8 + * [ tail_call_cnt ] 8 + * [ local_tmp_var ] 8 + * [ unused red zone ] 208 bytes protected + */ +static int bpf_jit_stack_local(struct codegen_context *ctx) +{ + if (bpf_has_stack_frame(ctx)) + return STACK_FRAME_MIN_SIZE + MAX_BPF_STACK; + else + return -(BPF_PPC_STACK_SAVE + 16); +} + +static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx) +{ + return bpf_jit_stack_local(ctx) + 8; +} + +static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg) +{ + if (reg >= BPF_PPC_NVR_MIN && reg < 32) + return (bpf_has_stack_frame(ctx) ? BPF_PPC_STACKFRAME : 0) + - (8 * (32 - reg)); + + pr_err("BPF JIT is asking about unknown registers"); + BUG(); +} + static void bpf_jit_emit_skb_loads(u32 *image, struct codegen_context *ctx) { /* @@ -73,36 +108,27 @@ static void bpf_jit_emit_skb_loads(u32 *image, struct codegen_context *ctx) PPC_BPF_LL(b2p[SKB_DATA_REG], 3, offsetof(struct sk_buff, data)); } -static void bpf_jit_emit_func_call(u32 *image, struct codegen_context *ctx, u64 func) +static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) { -#ifdef PPC64_ELF_ABI_v1 - /* func points to the function descriptor */ - PPC_LI64(b2p[TMP_REG_2], func); - /* Load actual entry point from function descriptor */ - PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0); - /* ... and move it to LR */ - PPC_MTLR(b2p[TMP_REG_1]); + int i; + /* - * Load TOC from function descriptor at offset 8. - * We can clobber r2 since we get called through a - * function pointer (so caller will save/restore r2) - * and since we don't use a TOC ourself. + * Initialize tail_call_cnt if we do tail calls. + * Otherwise, put in NOPs so that it can be skipped when we are + * invoked through a tail call. */ - PPC_BPF_LL(2, b2p[TMP_REG_2], 8); -#else - /* We can clobber r12 */ - PPC_FUNC_ADDR(12, func); - PPC_MTLR(12); -#endif - PPC_BLRL(); -} + if (ctx->seen & SEEN_TAILCALL) { + PPC_LI(b2p[TMP_REG_1], 0); + /* this goes in the redzone */ + PPC_BPF_STL(b2p[TMP_REG_1], 1, -(BPF_PPC_STACK_SAVE + 8)); + } else { + PPC_NOP(); + PPC_NOP(); + } -static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) -{ - int i; - bool new_stack_frame = bpf_has_stack_frame(ctx); +#define BPF_TAILCALL_PROLOGUE_SIZE 8 - if (new_stack_frame) { + if (bpf_has_stack_frame(ctx)) { /* * We need a stack frame, but we don't necessarily need to * save/restore LR unless we call other functions @@ -122,9 +148,7 @@ static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) */ for (i = BPF_REG_6; i <= BPF_REG_10; i++) if (bpf_is_seen_register(ctx, i)) - PPC_BPF_STL(b2p[i], 1, - (new_stack_frame ? BPF_PPC_STACKFRAME : 0) - - (8 * (32 - b2p[i]))); + PPC_BPF_STL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])); /* * Save additional non-volatile regs if we cache skb @@ -132,53 +156,142 @@ static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) */ if (ctx->seen & SEEN_SKB) { PPC_BPF_STL(b2p[SKB_HLEN_REG], 1, - BPF_PPC_STACKFRAME - (8 * (32 - b2p[SKB_HLEN_REG]))); + bpf_jit_stack_offsetof(ctx, b2p[SKB_HLEN_REG])); PPC_BPF_STL(b2p[SKB_DATA_REG], 1, - BPF_PPC_STACKFRAME - (8 * (32 - b2p[SKB_DATA_REG]))); + bpf_jit_stack_offsetof(ctx, b2p[SKB_DATA_REG])); bpf_jit_emit_skb_loads(image, ctx); } /* Setup frame pointer to point to the bpf stack area */ if (bpf_is_seen_register(ctx, BPF_REG_FP)) PPC_ADDI(b2p[BPF_REG_FP], 1, - BPF_PPC_STACKFRAME - BPF_PPC_STACK_SAVE); + STACK_FRAME_MIN_SIZE + MAX_BPF_STACK); } -static void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) +static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx) { int i; - bool new_stack_frame = bpf_has_stack_frame(ctx); - - /* Move result to r3 */ - PPC_MR(3, b2p[BPF_REG_0]); /* Restore NVRs */ for (i = BPF_REG_6; i <= BPF_REG_10; i++) if (bpf_is_seen_register(ctx, i)) - PPC_BPF_LL(b2p[i], 1, - (new_stack_frame ? BPF_PPC_STACKFRAME : 0) - - (8 * (32 - b2p[i]))); + PPC_BPF_LL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])); /* Restore non-volatile registers used for skb cache */ if (ctx->seen & SEEN_SKB) { PPC_BPF_LL(b2p[SKB_HLEN_REG], 1, - BPF_PPC_STACKFRAME - (8 * (32 - b2p[SKB_HLEN_REG]))); + bpf_jit_stack_offsetof(ctx, b2p[SKB_HLEN_REG])); PPC_BPF_LL(b2p[SKB_DATA_REG], 1, - BPF_PPC_STACKFRAME - (8 * (32 - b2p[SKB_DATA_REG]))); + bpf_jit_stack_offsetof(ctx, b2p[SKB_DATA_REG])); } /* Tear down our stack frame */ - if (new_stack_frame) { + if (bpf_has_stack_frame(ctx)) { PPC_ADDI(1, 1, BPF_PPC_STACKFRAME); if (ctx->seen & SEEN_FUNC) { PPC_BPF_LL(0, 1, PPC_LR_STKOFF); PPC_MTLR(0); } } +} + +static void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) +{ + bpf_jit_emit_common_epilogue(image, ctx); + + /* Move result to r3 */ + PPC_MR(3, b2p[BPF_REG_0]); PPC_BLR(); } +static void bpf_jit_emit_func_call(u32 *image, struct codegen_context *ctx, u64 func) +{ +#ifdef PPC64_ELF_ABI_v1 + /* func points to the function descriptor */ + PPC_LI64(b2p[TMP_REG_2], func); + /* Load actual entry point from function descriptor */ + PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0); + /* ... and move it to LR */ + PPC_MTLR(b2p[TMP_REG_1]); + /* + * Load TOC from function descriptor at offset 8. + * We can clobber r2 since we get called through a + * function pointer (so caller will save/restore r2) + * and since we don't use a TOC ourself. + */ + PPC_BPF_LL(2, b2p[TMP_REG_2], 8); +#else + /* We can clobber r12 */ + PPC_FUNC_ADDR(12, func); + PPC_MTLR(12); +#endif + PPC_BLRL(); +} + +static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out) +{ + /* + * By now, the eBPF program has already setup parameters in r3, r4 and r5 + * r3/BPF_REG_1 - pointer to ctx -- passed as is to the next bpf program + * r4/BPF_REG_2 - pointer to bpf_array + * r5/BPF_REG_3 - index in bpf_array + */ + int b2p_bpf_array = b2p[BPF_REG_2]; + int b2p_index = b2p[BPF_REG_3]; + + /* + * if (index >= array->map.max_entries) + * goto out; + */ + PPC_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)); + PPC_CMPLW(b2p_index, b2p[TMP_REG_1]); + PPC_BCC(COND_GE, out); + + /* + * if (tail_call_cnt > MAX_TAIL_CALL_CNT) + * goto out; + */ + PPC_LD(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)); + PPC_CMPLWI(b2p[TMP_REG_1], MAX_TAIL_CALL_CNT); + PPC_BCC(COND_GT, out); + + /* + * tail_call_cnt++; + */ + PPC_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], 1); + PPC_BPF_STL(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)); + + /* prog = array->ptrs[index]; */ + PPC_MULI(b2p[TMP_REG_1], b2p_index, 8); + PPC_ADD(b2p[TMP_REG_1], b2p[TMP_REG_1], b2p_bpf_array); + PPC_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_array, ptrs)); + + /* + * if (prog == NULL) + * goto out; + */ + PPC_CMPLDI(b2p[TMP_REG_1], 0); + PPC_BCC(COND_EQ, out); + + /* goto *(prog->bpf_func + prologue_size); */ + PPC_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func)); +#ifdef PPC64_ELF_ABI_v1 + /* skip past the function descriptor */ + PPC_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], + FUNCTION_DESCR_SIZE + BPF_TAILCALL_PROLOGUE_SIZE); +#else + PPC_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], BPF_TAILCALL_PROLOGUE_SIZE); +#endif + PPC_MTCTR(b2p[TMP_REG_1]); + + /* tear down stack, restore NVRs, ... */ + bpf_jit_emit_common_epilogue(image, ctx); + + PPC_BCTR(); + /* out: */ +} + /* Assemble the body code between the prologue & epilogue */ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx, @@ -200,7 +313,6 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u64 imm64; u8 *func; u32 true_cond; - int stack_local_off; /* * addrs[] maps a BPF bytecode address into a real offset from @@ -219,9 +331,9 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, * optimization but everything else should work without * any issues. */ - if (dst_reg >= 24 && dst_reg <= 31) + if (dst_reg >= BPF_PPC_NVR_MIN && dst_reg < 32) bpf_set_seen_register(ctx, insn[i].dst_reg); - if (src_reg >= 24 && src_reg <= 31) + if (src_reg >= BPF_PPC_NVR_MIN && src_reg < 32) bpf_set_seen_register(ctx, insn[i].src_reg); switch (code) { @@ -490,25 +602,12 @@ bpf_alu32_trunc: * Way easier and faster(?) to store the value * into stack and then use ldbrx * - * First, determine where in stack we can store - * this: - * - if we have allotted a stack frame, then we - * will utilize the area set aside by - * BPF_PPC_STACK_LOCALS - * - else, we use the area beneath the NV GPR - * save area - * * ctx->seen will be reliable in pass2, but * the instructions generated will remain the * same across all passes */ - if (bpf_has_stack_frame(ctx)) - stack_local_off = STACK_FRAME_MIN_SIZE; - else - stack_local_off = -(BPF_PPC_STACK_SAVE + 8); - - PPC_STD(dst_reg, 1, stack_local_off); - PPC_ADDI(b2p[TMP_REG_1], 1, stack_local_off); + PPC_STD(dst_reg, 1, bpf_jit_stack_local(ctx)); + PPC_ADDI(b2p[TMP_REG_1], 1, bpf_jit_stack_local(ctx)); PPC_LDBRX(dst_reg, 0, b2p[TMP_REG_1]); break; } @@ -668,7 +767,7 @@ emit_clear: /* Save skb pointer if we need to re-cache skb data */ if (bpf_helper_changes_skb_data(func)) - PPC_BPF_STL(3, 1, STACK_FRAME_MIN_SIZE); + PPC_BPF_STL(3, 1, bpf_jit_stack_local(ctx)); bpf_jit_emit_func_call(image, ctx, (u64)func); @@ -678,7 +777,7 @@ emit_clear: /* refresh skb cache */ if (bpf_helper_changes_skb_data(func)) { /* reload skb pointer to r3 */ - PPC_BPF_LL(3, 1, STACK_FRAME_MIN_SIZE); + PPC_BPF_LL(3, 1, bpf_jit_stack_local(ctx)); bpf_jit_emit_skb_loads(image, ctx); } break; @@ -837,9 +936,12 @@ common_load: break; /* - * TODO: Tail call + * Tail call */ case BPF_JMP | BPF_CALL | BPF_X: + ctx->seen |= SEEN_TAILCALL; + bpf_jit_emit_tail_call(image, ctx, addrs[i + 1]); + break; default: /* @@ -872,21 +974,37 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) int pass; int flen; struct bpf_binary_header *bpf_hdr; + struct bpf_prog *org_fp = fp; + struct bpf_prog *tmp_fp; + bool bpf_blinded = false; if (!bpf_jit_enable) - return fp; + return org_fp; + + tmp_fp = bpf_jit_blind_constants(org_fp); + if (IS_ERR(tmp_fp)) + return org_fp; + + if (tmp_fp != org_fp) { + bpf_blinded = true; + fp = tmp_fp; + } flen = fp->len; addrs = kzalloc((flen+1) * sizeof(*addrs), GFP_KERNEL); - if (addrs == NULL) - return fp; + if (addrs == NULL) { + fp = org_fp; + goto out; + } + + memset(&cgctx, 0, sizeof(struct codegen_context)); - cgctx.idx = 0; - cgctx.seen = 0; /* Scouting faux-generate pass 0 */ - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) + if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) { /* We hit something illegal or unsupported. */ + fp = org_fp; goto out; + } /* * Pretend to build prologue, given the features we've seen. This will @@ -901,8 +1019,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) bpf_hdr = bpf_jit_binary_alloc(alloclen, &image, 4, bpf_jit_fill_ill_insns); - if (!bpf_hdr) + if (!bpf_hdr) { + fp = org_fp; goto out; + } code_base = (u32 *)(image + FUNCTION_DESCR_SIZE); @@ -939,6 +1059,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) out: kfree(addrs); + + if (bpf_blinded) + bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp); + return fp; } diff --git a/arch/powerpc/oprofile/cell/spu_profiler.c b/arch/powerpc/oprofile/cell/spu_profiler.c index b129d007e7fe..b19265de9178 100644 --- a/arch/powerpc/oprofile/cell/spu_profiler.c +++ b/arch/powerpc/oprofile/cell/spu_profiler.c @@ -43,7 +43,7 @@ static unsigned int profiling_interval; #define SPU_PC_MASK 0xFFFF DEFINE_SPINLOCK(oprof_spu_smpl_arry_lck); -unsigned long oprof_spu_smpl_arry_lck_flags; +static unsigned long oprof_spu_smpl_arry_lck_flags; void set_spu_profiling_frequency(unsigned int freq_khz, unsigned int cycles_reset) { diff --git a/arch/powerpc/oprofile/cell/spu_task_sync.c b/arch/powerpc/oprofile/cell/spu_task_sync.c index ef2142ff7dbd..83d2b4ef7f0d 100644 --- a/arch/powerpc/oprofile/cell/spu_task_sync.c +++ b/arch/powerpc/oprofile/cell/spu_task_sync.c @@ -36,7 +36,7 @@ static DEFINE_SPINLOCK(buffer_lock); static DEFINE_SPINLOCK(cache_lock); static int num_spu_nodes; -int spu_prof_num_nodes; +static int spu_prof_num_nodes; struct spu_buffer spu_buff[MAX_NUMNODES * SPUS_PER_NODE]; struct delayed_work spu_work; @@ -88,7 +88,7 @@ static void spu_buff_add(unsigned long int value, int spu) /* This function copies the per SPU buffers to the * OProfile kernel buffer. */ -void sync_spu_buff(void) +static void sync_spu_buff(void) { int spu; unsigned long flags; diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c index 4ed377f0f7b2..72c27b8d2cf3 100644 --- a/arch/powerpc/perf/core-book3s.c +++ b/arch/powerpc/perf/core-book3s.c @@ -2158,7 +2158,7 @@ static void perf_event_interrupt(struct pt_regs *regs) irq_exit(); } -int power_pmu_prepare_cpu(unsigned int cpu) +static int power_pmu_prepare_cpu(unsigned int cpu) { struct cpu_hw_events *cpuhw = &per_cpu(cpu_hw_events, cpu); diff --git a/arch/powerpc/perf/hv-gpci.c b/arch/powerpc/perf/hv-gpci.c index 7aa37236bb70..43fabb3cae0f 100644 --- a/arch/powerpc/perf/hv-gpci.c +++ b/arch/powerpc/perf/hv-gpci.c @@ -131,7 +131,7 @@ static const struct attribute_group *attr_groups[] = { #define HGPCI_MAX_DATA_BYTES \ (HGPCI_REQ_BUFFER_SIZE - sizeof(struct hv_get_perf_counter_info_params)) -DEFINE_PER_CPU(char, hv_gpci_reqb[HGPCI_REQ_BUFFER_SIZE]) __aligned(sizeof(uint64_t)); +static DEFINE_PER_CPU(char, hv_gpci_reqb[HGPCI_REQ_BUFFER_SIZE]) __aligned(sizeof(uint64_t)); struct hv_gpci_request_buffer { struct hv_get_perf_counter_info_params params; diff --git a/arch/powerpc/perf/power7-pmu.c b/arch/powerpc/perf/power7-pmu.c index a383c23a9070..7963658dbc22 100644 --- a/arch/powerpc/perf/power7-pmu.c +++ b/arch/powerpc/perf/power7-pmu.c @@ -416,7 +416,7 @@ static struct attribute *power7_pmu_format_attr[] = { NULL, }; -struct attribute_group power7_pmu_format_group = { +static struct attribute_group power7_pmu_format_group = { .name = "format", .attrs = power7_pmu_format_attr, }; diff --git a/arch/powerpc/perf/power8-pmu.c b/arch/powerpc/perf/power8-pmu.c index 5fde2b192fec..ab830d106ec5 100644 --- a/arch/powerpc/perf/power8-pmu.c +++ b/arch/powerpc/perf/power8-pmu.c @@ -204,7 +204,7 @@ static struct attribute *power8_pmu_format_attr[] = { NULL, }; -struct attribute_group power8_pmu_format_group = { +static struct attribute_group power8_pmu_format_group = { .name = "format", .attrs = power8_pmu_format_attr, }; diff --git a/arch/powerpc/perf/power9-pmu.c b/arch/powerpc/perf/power9-pmu.c index 788346303852..8e9a81967ff8 100644 --- a/arch/powerpc/perf/power9-pmu.c +++ b/arch/powerpc/perf/power9-pmu.c @@ -119,7 +119,7 @@ static struct attribute *power9_pmu_format_attr[] = { NULL, }; -struct attribute_group power9_pmu_format_group = { +static struct attribute_group power9_pmu_format_group = { .name = "format", .attrs = power9_pmu_format_attr, }; diff --git a/arch/powerpc/platforms/44x/warp.c b/arch/powerpc/platforms/44x/warp.c index 5ecce543103e..a886c2c22097 100644 --- a/arch/powerpc/platforms/44x/warp.c +++ b/arch/powerpc/platforms/44x/warp.c @@ -204,7 +204,7 @@ static void pika_setup_critical_temp(struct device_node *np, i2c_smbus_write_byte_data(client, 3, 0); /* Tlow */ irq = irq_of_parse_and_map(np, 0); - if (irq == NO_IRQ) { + if (!irq) { printk(KERN_ERR __FILE__ ": Unable to get ad7414 irq\n"); return; } diff --git a/arch/powerpc/platforms/512x/mpc5121_ads_cpld.c b/arch/powerpc/platforms/512x/mpc5121_ads_cpld.c index 0035d146df73..fe4d4eac7427 100644 --- a/arch/powerpc/platforms/512x/mpc5121_ads_cpld.c +++ b/arch/powerpc/platforms/512x/mpc5121_ads_cpld.c @@ -97,7 +97,7 @@ cpld_pic_get_irq(int offset, u8 ignore, u8 __iomem *statusp, status |= (ignore | mask); if (status == 0xff) - return NO_IRQ; + return 0; cpld_irq = ffz(status) + offset; @@ -110,14 +110,14 @@ static void cpld_pic_cascade(struct irq_desc *desc) irq = cpld_pic_get_irq(0, PCI_IGNORE, &cpld_regs->pci_status, &cpld_regs->pci_mask); - if (irq != NO_IRQ) { + if (irq) { generic_handle_irq(irq); return; } irq = cpld_pic_get_irq(8, MISC_IGNORE, &cpld_regs->misc_status, &cpld_regs->misc_mask); - if (irq != NO_IRQ) { + if (irq) { generic_handle_irq(irq); return; } @@ -177,7 +177,7 @@ mpc5121_ads_cpld_pic_init(void) goto end; cascade_irq = irq_of_parse_and_map(np, 0); - if (cascade_irq == NO_IRQ) + if (!cascade_irq) goto end; /* diff --git a/arch/powerpc/platforms/512x/mpc512x_lpbfifo.c b/arch/powerpc/platforms/512x/mpc512x_lpbfifo.c index d93dd4acf40b..cec3f88f153d 100644 --- a/arch/powerpc/platforms/512x/mpc512x_lpbfifo.c +++ b/arch/powerpc/platforms/512x/mpc512x_lpbfifo.c @@ -473,7 +473,7 @@ static int mpc512x_lpbfifo_probe(struct platform_device *pdev) } lpbfifo.irq = irq_of_parse_and_map(pdev->dev.of_node, 0); - if (lpbfifo.irq == NO_IRQ) { + if (!lpbfifo.irq) { dev_err(&pdev->dev, "mapping irq failed\n"); ret = -ENODEV; goto err0; diff --git a/arch/powerpc/platforms/52xx/mpc52xx_pic.c b/arch/powerpc/platforms/52xx/mpc52xx_pic.c index 4fe2074c88cb..fc98912f42cf 100644 --- a/arch/powerpc/platforms/52xx/mpc52xx_pic.c +++ b/arch/powerpc/platforms/52xx/mpc52xx_pic.c @@ -511,7 +511,7 @@ unsigned int mpc52xx_get_irq(void) irq |= (MPC52xx_IRQ_L1_PERP << MPC52xx_IRQ_L1_OFFSET); } } else { - return NO_IRQ; + return 0; } return irq_linear_revmap(mpc52xx_irqhost, irq); diff --git a/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c b/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c index 60e89fc9c753..8b065bdf7412 100644 --- a/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c +++ b/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c @@ -131,7 +131,7 @@ int __init pq2ads_pci_init_irq(void) } irq = irq_of_parse_and_map(np, 0); - if (irq == NO_IRQ) { + if (!irq) { printk(KERN_ERR "No interrupt in pci pic node.\n"); of_node_put(np); goto out; diff --git a/arch/powerpc/platforms/83xx/mpc832x_rdb.c b/arch/powerpc/platforms/83xx/mpc832x_rdb.c index 2ef03e7d248c..0d6a62fc5864 100644 --- a/arch/powerpc/platforms/83xx/mpc832x_rdb.c +++ b/arch/powerpc/platforms/83xx/mpc832x_rdb.c @@ -89,7 +89,7 @@ static int __init of_fsl_spi_probe(char *type, char *compatible, u32 sysclk, goto err; ret = of_irq_to_resource(np, 0, &res[1]); - if (ret == NO_IRQ) + if (!ret) goto err; pdev = platform_device_alloc("mpc83xx_spi", i); diff --git a/arch/powerpc/platforms/83xx/suspend.c b/arch/powerpc/platforms/83xx/suspend.c index fcbea4b51a78..24717d060008 100644 --- a/arch/powerpc/platforms/83xx/suspend.c +++ b/arch/powerpc/platforms/83xx/suspend.c @@ -352,7 +352,7 @@ static int pmc_probe(struct platform_device *ofdev) return -ENODEV; pmc_irq = irq_of_parse_and_map(np, 0); - if (pmc_irq != NO_IRQ) { + if (pmc_irq) { ret = request_irq(pmc_irq, pmc_irq_handler, IRQF_SHARED, "pmc", ofdev); @@ -400,7 +400,7 @@ out_syscr: out_pmc: iounmap(pmc_regs); out: - if (pmc_irq != NO_IRQ) + if (pmc_irq) free_irq(pmc_irq, ofdev); return ret; diff --git a/arch/powerpc/platforms/85xx/common.c b/arch/powerpc/platforms/85xx/common.c index 28720a4ded7b..954e5e8b14ef 100644 --- a/arch/powerpc/platforms/85xx/common.c +++ b/arch/powerpc/platforms/85xx/common.c @@ -76,7 +76,7 @@ void __init mpc85xx_cpm2_pic_init(void) return; } irq = irq_of_parse_and_map(np, 0); - if (irq == NO_IRQ) { + if (!irq) { of_node_put(np); printk(KERN_ERR "PIC init: got no IRQ for cpm cascade\n"); return; diff --git a/arch/powerpc/platforms/85xx/mpc85xx_cds.c b/arch/powerpc/platforms/85xx/mpc85xx_cds.c index 62f171c71c4c..86f20156178e 100644 --- a/arch/powerpc/platforms/85xx/mpc85xx_cds.c +++ b/arch/powerpc/platforms/85xx/mpc85xx_cds.c @@ -196,7 +196,7 @@ static void mpc85xx_8259_cascade_handler(struct irq_desc *desc) { unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) /* handle an interrupt from the 8259 */ generic_handle_irq(cascade_irq); @@ -247,7 +247,7 @@ static int mpc85xx_cds_8259_attach(void) } cascade_irq = irq_of_parse_and_map(cascade_node, 0); - if (cascade_irq == NO_IRQ) { + if (!cascade_irq) { printk(KERN_ERR "Failed to map cascade interrupt\n"); return -ENXIO; } diff --git a/arch/powerpc/platforms/85xx/mpc85xx_ds.c b/arch/powerpc/platforms/85xx/mpc85xx_ds.c index 6bc07d837b1c..ed69c7ee1829 100644 --- a/arch/powerpc/platforms/85xx/mpc85xx_ds.c +++ b/arch/powerpc/platforms/85xx/mpc85xx_ds.c @@ -51,7 +51,7 @@ static void mpc85xx_8259_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) { + if (cascade_irq) { generic_handle_irq(cascade_irq); } chip->irq_eoi(&desc->irq_data); @@ -96,7 +96,7 @@ void __init mpc85xx_ds_pic_init(void) } cascade_irq = irq_of_parse_and_map(cascade_node, 0); - if (cascade_irq == NO_IRQ) { + if (!cascade_irq) { printk(KERN_ERR "Failed to map cascade interrupt\n"); return; } diff --git a/arch/powerpc/platforms/85xx/socrates_fpga_pic.c b/arch/powerpc/platforms/85xx/socrates_fpga_pic.c index b02d6a5bb035..82f8490b5aa7 100644 --- a/arch/powerpc/platforms/85xx/socrates_fpga_pic.c +++ b/arch/powerpc/platforms/85xx/socrates_fpga_pic.c @@ -78,7 +78,7 @@ static inline unsigned int socrates_fpga_pic_get_irq(unsigned int irq) break; } if (i == 3) - return NO_IRQ; + return 0; raw_spin_lock_irqsave(&socrates_fpga_pic_lock, flags); cause = socrates_fpga_pic_read(FPGA_PIC_IRQMASK(i)); @@ -103,7 +103,7 @@ static void socrates_fpga_pic_cascade(struct irq_desc *desc) */ cascade_irq = socrates_fpga_pic_get_irq(irq); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); } @@ -292,7 +292,7 @@ void socrates_fpga_pic_init(struct device_node *pic) for (i = 0; i < 3; i++) { socrates_fpga_irqs[i] = irq_of_parse_and_map(pic, i); - if (socrates_fpga_irqs[i] == NO_IRQ) { + if (!socrates_fpga_irqs[i]) { pr_warning("FPGA PIC: can't get irq%d.\n", i); continue; } diff --git a/arch/powerpc/platforms/86xx/pic.c b/arch/powerpc/platforms/86xx/pic.c index 845defa1fd19..a6c695fa4da0 100644 --- a/arch/powerpc/platforms/86xx/pic.c +++ b/arch/powerpc/platforms/86xx/pic.c @@ -22,7 +22,7 @@ static void mpc86xx_8259_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); @@ -58,7 +58,7 @@ void __init mpc86xx_init_irq(void) } cascade_irq = irq_of_parse_and_map(cascade_node, 0); - if (cascade_irq == NO_IRQ) { + if (!cascade_irq) { printk(KERN_ERR "Failed to map cascade interrupt\n"); return; } diff --git a/arch/powerpc/platforms/8xx/m8xx_setup.c b/arch/powerpc/platforms/8xx/m8xx_setup.c index b1ab6e96cb31..f81069f79a94 100644 --- a/arch/powerpc/platforms/8xx/m8xx_setup.c +++ b/arch/powerpc/platforms/8xx/m8xx_setup.c @@ -241,6 +241,6 @@ void __init mpc8xx_pics_init(void) } irq = cpm_pic_init(); - if (irq != NO_IRQ) + if (irq) irq_set_chained_handler(irq, cpm_cascade); } diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index f32edec13fd1..ca2da30ad2ab 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -74,6 +74,7 @@ config PPC_BOOK3S_64 select HAVE_ARCH_TRANSPARENT_HUGEPAGE select ARCH_SUPPORTS_NUMA_BALANCING select IRQ_WORK + select HAVE_KERNEL_XZ config PPC_BOOK3E_64 bool "Embedded processors" @@ -86,6 +87,7 @@ endchoice choice prompt "CPU selection" depends on PPC64 + default POWER8_CPU if CPU_LITTLE_ENDIAN default GENERIC_CPU help This will create a kernel which is optimised for a particular CPU. diff --git a/arch/powerpc/platforms/cell/axon_msi.c b/arch/powerpc/platforms/cell/axon_msi.c index aed7714495c1..8b55c5f19d4c 100644 --- a/arch/powerpc/platforms/cell/axon_msi.c +++ b/arch/powerpc/platforms/cell/axon_msi.c @@ -271,7 +271,7 @@ static int axon_msi_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) for_each_pci_msi_entry(entry, dev) { virq = irq_create_direct_mapping(msic->irq_domain); - if (virq == NO_IRQ) { + if (!virq) { dev_warn(&dev->dev, "axon_msi: virq allocation failed!\n"); return -1; @@ -293,7 +293,7 @@ static void axon_msi_teardown_msi_irqs(struct pci_dev *dev) dev_dbg(&dev->dev, "axon_msi: tearing down msi irqs\n"); for_each_pci_msi_entry(entry, dev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; irq_set_msi_desc(entry->irq, NULL); @@ -375,7 +375,7 @@ static int axon_msi_probe(struct platform_device *device) } virq = irq_of_parse_and_map(dn, 0); - if (virq == NO_IRQ) { + if (!virq) { printk(KERN_ERR "axon_msi: irq parse and map failed for %s\n", dn->full_name); goto out_free_fifo; diff --git a/arch/powerpc/platforms/cell/cbe_regs.c b/arch/powerpc/platforms/cell/cbe_regs.c index 1428d583c238..b926438d73af 100644 --- a/arch/powerpc/platforms/cell/cbe_regs.c +++ b/arch/powerpc/platforms/cell/cbe_regs.c @@ -189,7 +189,7 @@ static struct device_node *cbe_get_be_node(int cpu_id) return NULL; } -void __init cbe_fill_regs_map(struct cbe_regs_map *map) +static void __init cbe_fill_regs_map(struct cbe_regs_map *map) { if(map->be_node) { struct device_node *be, *np; diff --git a/arch/powerpc/platforms/cell/interrupt.c b/arch/powerpc/platforms/cell/interrupt.c index 9f609fc8d331..a6bbbaba14a3 100644 --- a/arch/powerpc/platforms/cell/interrupt.c +++ b/arch/powerpc/platforms/cell/interrupt.c @@ -123,7 +123,7 @@ static void iic_ioexc_cascade(struct irq_desc *desc) unsigned int cirq = irq_linear_revmap(iic_host, base | cascade); - if (cirq != NO_IRQ) + if (cirq) generic_handle_irq(cirq); } /* post-ack level interrupts */ @@ -153,10 +153,10 @@ static unsigned int iic_get_irq(void) *(unsigned long *) &pending = in_be64((u64 __iomem *) &iic->regs->pending_destr); if (!(pending.flags & CBE_IIC_IRQ_VALID)) - return NO_IRQ; + return 0; virq = irq_linear_revmap(iic_host, iic_pending_to_hwnum(pending)); - if (virq == NO_IRQ) - return NO_IRQ; + if (!virq) + return 0; iic->eoi_stack[++iic->eoi_ptr] = pending.prio; BUG_ON(iic->eoi_ptr > 15); return virq; @@ -187,18 +187,12 @@ void iic_message_pass(int cpu, int msg) out_be64(&per_cpu(cpu_iic, cpu).regs->generate, (0xf - msg) << 4); } -struct irq_domain *iic_get_irq_host(int node) -{ - return iic_host; -} -EXPORT_SYMBOL_GPL(iic_get_irq_host); - static void iic_request_ipi(int msg) { int virq; virq = irq_create_mapping(iic_host, iic_msg_to_irq(msg)); - if (virq == NO_IRQ) { + if (!virq) { printk(KERN_ERR "iic: failed to map IPI %s\n", smp_ipi_name[msg]); return; @@ -353,7 +347,7 @@ static int __init setup_iic(void) cascade |= 1 << IIC_IRQ_CLASS_SHIFT; cascade |= IIC_UNIT_IIC; cascade = irq_create_mapping(iic_host, cascade); - if (cascade == NO_IRQ) + if (!cascade) continue; /* * irq_data is a generic pointer that gets passed back diff --git a/arch/powerpc/platforms/cell/iommu.c b/arch/powerpc/platforms/cell/iommu.c index f7d1a4953ea0..7ff51f96a00e 100644 --- a/arch/powerpc/platforms/cell/iommu.c +++ b/arch/powerpc/platforms/cell/iommu.c @@ -411,7 +411,7 @@ static void cell_iommu_enable_hardware(struct cbe_iommu *iommu) virq = irq_create_mapping(NULL, IIC_IRQ_IOEX_ATI | (iommu->nid << IIC_IRQ_NODE_SHIFT)); - BUG_ON(virq == NO_IRQ); + BUG_ON(!virq); ret = request_irq(virq, ioc_interrupt, 0, iommu->name, iommu); BUG_ON(ret); @@ -651,7 +651,7 @@ static int dma_fixed_dma_supported(struct device *dev, u64 mask) static int dma_set_mask_and_switch(struct device *dev, u64 dma_mask); -struct dma_map_ops dma_iommu_fixed_ops = { +static struct dma_map_ops dma_iommu_fixed_ops = { .alloc = dma_fixed_alloc_coherent, .free = dma_fixed_free_coherent, .map_sg = dma_fixed_map_sg, diff --git a/arch/powerpc/platforms/cell/pmu.c b/arch/powerpc/platforms/cell/pmu.c index 348a27b12512..e3ad0c38f017 100644 --- a/arch/powerpc/platforms/cell/pmu.c +++ b/arch/powerpc/platforms/cell/pmu.c @@ -385,7 +385,7 @@ static int __init cbe_init_pm_irq(void) for_each_online_node(node) { irq = irq_create_mapping(NULL, IIC_IRQ_IOEX_PMI | (node << IIC_IRQ_NODE_SHIFT)); - if (irq == NO_IRQ) { + if (!irq) { printk("ERROR: Unable to allocate irq for node %d\n", node); return -EINVAL; @@ -412,7 +412,7 @@ void cbe_sync_irq(int node) IIC_IRQ_IOEX_PMI | (node << IIC_IRQ_NODE_SHIFT)); - if (irq == NO_IRQ) { + if (!irq) { printk(KERN_WARNING "ERROR, unable to get existing irq %d " \ "for node %d\n", irq, node); return; diff --git a/arch/powerpc/platforms/cell/ras.c b/arch/powerpc/platforms/cell/ras.c index 2d4f60c0119a..460ab392f0e7 100644 --- a/arch/powerpc/platforms/cell/ras.c +++ b/arch/powerpc/platforms/cell/ras.c @@ -298,7 +298,7 @@ int cbe_sysreset_hack(void) } #endif /* CONFIG_PPC_IBM_CELL_RESETBUTTON */ -int __init cbe_ptcal_init(void) +static int __init cbe_ptcal_init(void) { int ret; ptcal_start_tok = rtas_token("ibm,cbe-start-ptcal"); diff --git a/arch/powerpc/platforms/cell/spider-pic.c b/arch/powerpc/platforms/cell/spider-pic.c index d06dcac66fcb..ff924af00e78 100644 --- a/arch/powerpc/platforms/cell/spider-pic.c +++ b/arch/powerpc/platforms/cell/spider-pic.c @@ -207,11 +207,11 @@ static void spider_irq_cascade(struct irq_desc *desc) cs = in_be32(pic->regs + TIR_CS) >> 24; if (cs == SPIDER_IRQ_INVALID) - virq = NO_IRQ; + virq = 0; else virq = irq_linear_revmap(pic->host, cs); - if (virq != NO_IRQ) + if (virq) generic_handle_irq(virq); chip->irq_eoi(&desc->irq_data); @@ -245,19 +245,19 @@ static unsigned int __init spider_find_cascade_and_node(struct spider_pic *pic) /* Now do the horrible hacks */ tmp = of_get_property(of_node, "#interrupt-cells", NULL); if (tmp == NULL) - return NO_IRQ; + return 0; intsize = *tmp; imap = of_get_property(of_node, "interrupt-map", &imaplen); if (imap == NULL || imaplen < (intsize + 1)) - return NO_IRQ; + return 0; iic = of_find_node_by_phandle(imap[intsize]); if (iic == NULL) - return NO_IRQ; + return 0; imap += intsize + 1; tmp = of_get_property(iic, "#interrupt-cells", NULL); if (tmp == NULL) { of_node_put(iic); - return NO_IRQ; + return 0; } intsize = *tmp; /* Assume unit is last entry of interrupt specifier */ @@ -266,7 +266,7 @@ static unsigned int __init spider_find_cascade_and_node(struct spider_pic *pic) tmp = of_get_property(iic, "ibm,interrupt-server-ranges", NULL); if (tmp == NULL) { of_node_put(iic); - return NO_IRQ; + return 0; } /* ugly as hell but works for now */ pic->node_id = (*tmp) >> 1; @@ -281,7 +281,7 @@ static unsigned int __init spider_find_cascade_and_node(struct spider_pic *pic) (pic->node_id << IIC_IRQ_NODE_SHIFT) | (2 << IIC_IRQ_CLASS_SHIFT) | unit); - if (virq == NO_IRQ) + if (!virq) printk(KERN_ERR "spider_pic: failed to map cascade !"); return virq; } @@ -318,7 +318,7 @@ static void __init spider_init_one(struct device_node *of_node, int chip, /* Hook up the cascade interrupt to the iic and nodeid */ virq = spider_find_cascade_and_node(pic); - if (virq == NO_IRQ) + if (!virq) return; irq_set_handler_data(virq, pic); irq_set_chained_handler(virq, spider_irq_cascade); diff --git a/arch/powerpc/platforms/cell/spu_base.c b/arch/powerpc/platforms/cell/spu_base.c index bb4a8e07c229..e84d8fbc2e21 100644 --- a/arch/powerpc/platforms/cell/spu_base.c +++ b/arch/powerpc/platforms/cell/spu_base.c @@ -402,7 +402,7 @@ static int spu_request_irqs(struct spu *spu) { int ret = 0; - if (spu->irqs[0] != NO_IRQ) { + if (spu->irqs[0]) { snprintf(spu->irq_c0, sizeof (spu->irq_c0), "spe%02d.0", spu->number); ret = request_irq(spu->irqs[0], spu_irq_class_0, @@ -410,7 +410,7 @@ static int spu_request_irqs(struct spu *spu) if (ret) goto bail0; } - if (spu->irqs[1] != NO_IRQ) { + if (spu->irqs[1]) { snprintf(spu->irq_c1, sizeof (spu->irq_c1), "spe%02d.1", spu->number); ret = request_irq(spu->irqs[1], spu_irq_class_1, @@ -418,7 +418,7 @@ static int spu_request_irqs(struct spu *spu) if (ret) goto bail1; } - if (spu->irqs[2] != NO_IRQ) { + if (spu->irqs[2]) { snprintf(spu->irq_c2, sizeof (spu->irq_c2), "spe%02d.2", spu->number); ret = request_irq(spu->irqs[2], spu_irq_class_2, @@ -429,10 +429,10 @@ static int spu_request_irqs(struct spu *spu) return 0; bail2: - if (spu->irqs[1] != NO_IRQ) + if (spu->irqs[1]) free_irq(spu->irqs[1], spu); bail1: - if (spu->irqs[0] != NO_IRQ) + if (spu->irqs[0]) free_irq(spu->irqs[0], spu); bail0: return ret; @@ -440,11 +440,11 @@ bail0: static void spu_free_irqs(struct spu *spu) { - if (spu->irqs[0] != NO_IRQ) + if (spu->irqs[0]) free_irq(spu->irqs[0], spu); - if (spu->irqs[1] != NO_IRQ) + if (spu->irqs[1]) free_irq(spu->irqs[1], spu); - if (spu->irqs[2] != NO_IRQ) + if (spu->irqs[2]) free_irq(spu->irqs[2], spu); } diff --git a/arch/powerpc/platforms/cell/spu_manage.c b/arch/powerpc/platforms/cell/spu_manage.c index 21b4bfb97200..672d310dcf14 100644 --- a/arch/powerpc/platforms/cell/spu_manage.c +++ b/arch/powerpc/platforms/cell/spu_manage.c @@ -105,7 +105,10 @@ static int __init spu_map_interrupts_old(struct spu *spu, spu->irqs[2] = irq_create_mapping(NULL, IIC_IRQ_CLASS_2 | isrc); /* Right now, we only fail if class 2 failed */ - return spu->irqs[2] == NO_IRQ ? -EINVAL : 0; + if (!spu->irqs[2]) + return -EINVAL; + + return 0; } static void __iomem * __init spu_map_prop_old(struct spu *spu, @@ -191,7 +194,7 @@ static int __init spu_map_interrupts(struct spu *spu, struct device_node *np) pr_debug(" irq %d no 0x%x on %s\n", i, oirq.args[0], oirq.np->full_name); spu->irqs[i] = irq_create_of_mapping(&oirq); - if (spu->irqs[i] == NO_IRQ) { + if (!spu->irqs[i]) { pr_debug("spu_new: failed to map it !\n"); goto err; } @@ -202,7 +205,7 @@ err: pr_debug("failed to map irq %x for spu %s\n", *oirq.args, spu->name); for (; i >= 0; i--) { - if (spu->irqs[i] != NO_IRQ) + if (spu->irqs[i]) irq_dispose_mapping(spu->irqs[i]); } return ret; diff --git a/arch/powerpc/platforms/chrp/setup.c b/arch/powerpc/platforms/chrp/setup.c index bfb300633dfe..0ce1b45f02a8 100644 --- a/arch/powerpc/platforms/chrp/setup.c +++ b/arch/powerpc/platforms/chrp/setup.c @@ -368,7 +368,7 @@ static void chrp_8259_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); @@ -514,7 +514,7 @@ static void __init chrp_find_8259(void) } if (chrp_mpic != NULL) { cascade_irq = irq_of_parse_and_map(pic, 0); - if (cascade_irq == NO_IRQ) + if (!cascade_irq) printk(KERN_ERR "i8259: failed to map cascade irq\n"); else irq_set_chained_handler(cascade_irq, diff --git a/arch/powerpc/platforms/embedded6xx/flipper-pic.c b/arch/powerpc/platforms/embedded6xx/flipper-pic.c index b7866e01483d..ade83829d5e8 100644 --- a/arch/powerpc/platforms/embedded6xx/flipper-pic.c +++ b/arch/powerpc/platforms/embedded6xx/flipper-pic.c @@ -181,7 +181,7 @@ unsigned int flipper_pic_get_irq(void) irq_status = in_be32(io_base + FLIPPER_ICR) & in_be32(io_base + FLIPPER_IMR); if (irq_status == 0) - return NO_IRQ; /* no more IRQs pending */ + return 0; /* no more IRQs pending */ irq = __ffs(irq_status); return irq_linear_revmap(flipper_irq_host, irq); diff --git a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c index 9b7975706bfc..89c54de88b7a 100644 --- a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c +++ b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c @@ -114,7 +114,7 @@ static unsigned int __hlwd_pic_get_irq(struct irq_domain *h) irq_status = in_be32(io_base + HW_BROADWAY_ICR) & in_be32(io_base + HW_BROADWAY_IMR); if (irq_status == 0) - return NO_IRQ; /* no more IRQs pending */ + return 0; /* no more IRQs pending */ irq = __ffs(irq_status); return irq_linear_revmap(h, irq); @@ -131,7 +131,7 @@ static void hlwd_pic_irq_cascade(struct irq_desc *desc) raw_spin_unlock(&desc->lock); virq = __hlwd_pic_get_irq(irq_domain); - if (virq != NO_IRQ) + if (virq) generic_handle_irq(virq); else pr_err("spurious interrupt!\n"); diff --git a/arch/powerpc/platforms/embedded6xx/mvme5100.c b/arch/powerpc/platforms/embedded6xx/mvme5100.c index ed7321d6772e..8e3590941960 100644 --- a/arch/powerpc/platforms/embedded6xx/mvme5100.c +++ b/arch/powerpc/platforms/embedded6xx/mvme5100.c @@ -47,7 +47,7 @@ static void mvme5100_8259_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); @@ -84,7 +84,7 @@ static void __init mvme5100_pic_init(void) } cirq = irq_of_parse_and_map(cp, 0); - if (cirq == NO_IRQ) { + if (!cirq) { pr_warn("mvme5100_pic_init: no cascade interrupt?\n"); return; } diff --git a/arch/powerpc/platforms/maple/pci.c b/arch/powerpc/platforms/maple/pci.c index a2f89e6326ce..a0589aac4163 100644 --- a/arch/powerpc/platforms/maple/pci.c +++ b/arch/powerpc/platforms/maple/pci.c @@ -552,7 +552,7 @@ void maple_pci_irq_fixup(struct pci_dev *dev) pci_bus_to_host(dev->bus) == u4_pcie) { printk(KERN_DEBUG "Fixup U4 PCIe IRQ\n"); dev->irq = irq_create_mapping(NULL, 1); - if (dev->irq != NO_IRQ) + if (dev->irq) irq_set_irq_type(dev->irq, IRQ_TYPE_LEVEL_LOW); } @@ -562,7 +562,7 @@ void maple_pci_irq_fixup(struct pci_dev *dev) if (dev->vendor == PCI_VENDOR_ID_AMD && dev->device == PCI_DEVICE_ID_AMD_8111_IDE && (dev->class & 5) != 5) { - dev->irq = NO_IRQ; + dev->irq = 0; } DBG(" <- maple_pci_irq_fixup\n"); @@ -648,7 +648,7 @@ int maple_pci_get_legacy_ide_irq(struct pci_dev *pdev, int channel) return defirq; } irq = irq_of_parse_and_map(np, channel & 0x1); - if (irq == NO_IRQ) { + if (!irq) { printk("Failed to map onboard IDE interrupt for channel %d\n", channel); return defirq; diff --git a/arch/powerpc/platforms/maple/setup.c b/arch/powerpc/platforms/maple/setup.c index 3c30c7a4534d..b7f937563827 100644 --- a/arch/powerpc/platforms/maple/setup.c +++ b/arch/powerpc/platforms/maple/setup.c @@ -156,7 +156,7 @@ static void __noreturn maple_halt(void) } #ifdef CONFIG_SMP -struct smp_ops_t maple_smp_ops = { +static struct smp_ops_t maple_smp_ops = { .probe = smp_mpic_probe, .message_pass = smp_mpic_message_pass, .kick_cpu = smp_generic_kick_cpu, @@ -176,7 +176,7 @@ static void __init maple_use_rtas_reboot_and_halt_if_present(void) } } -void __init maple_setup_arch(void) +static void __init maple_setup_arch(void) { /* init to some ~sane value until calibrate_delay() runs */ loops_per_jiffy = 50000000; diff --git a/arch/powerpc/platforms/pasemi/Kconfig b/arch/powerpc/platforms/pasemi/Kconfig index 00d4b28cbb60..c7f1dbe94de7 100644 --- a/arch/powerpc/platforms/pasemi/Kconfig +++ b/arch/powerpc/platforms/pasemi/Kconfig @@ -14,6 +14,16 @@ config PPC_PASEMI menu "PA Semi PWRficient options" depends on PPC_PASEMI +config PPC_PASEMI_NEMO + bool "Nemo motherboard Support" + depends on PPC_PASEMI + select PPC_I8259 + help + This option enables support for the 'Nemo' motherboard + used in A-Eons's Amigaone X1000. This consists of some + device tree patches and workarounds for the SB600 South + Bridge that provides SATA/USB/Audio. + config PPC_PASEMI_IOMMU bool "PA Semi IOMMU support" depends on PPC_PASEMI diff --git a/arch/powerpc/platforms/pasemi/gpio_mdio.c b/arch/powerpc/platforms/pasemi/gpio_mdio.c index ddf635000c6b..c23e60959aa8 100644 --- a/arch/powerpc/platforms/pasemi/gpio_mdio.c +++ b/arch/powerpc/platforms/pasemi/gpio_mdio.c @@ -306,7 +306,7 @@ static struct platform_driver gpio_mdio_driver = }, }; -int gpio_mdio_init(void) +static int gpio_mdio_init(void) { struct device_node *np; @@ -326,7 +326,7 @@ int gpio_mdio_init(void) } module_init(gpio_mdio_init); -void gpio_mdio_exit(void) +static void gpio_mdio_exit(void) { platform_driver_unregister(&gpio_mdio_driver); if (gpio_regs) diff --git a/arch/powerpc/platforms/pasemi/iommu.c b/arch/powerpc/platforms/pasemi/iommu.c index c61667e8bb06..e74adc4e7fd8 100644 --- a/arch/powerpc/platforms/pasemi/iommu.c +++ b/arch/powerpc/platforms/pasemi/iommu.c @@ -199,7 +199,7 @@ static void pci_dma_dev_setup_pasemi(struct pci_dev *dev) set_iommu_table_base(&dev->dev, &iommu_table_iobmap); } -int __init iob_init(struct device_node *dn) +static int __init iob_init(struct device_node *dn) { unsigned long tmp; u32 regword; diff --git a/arch/powerpc/platforms/pasemi/misc.c b/arch/powerpc/platforms/pasemi/misc.c index e0ab299763c1..8571e7bf78b6 100644 --- a/arch/powerpc/platforms/pasemi/misc.c +++ b/arch/powerpc/platforms/pasemi/misc.c @@ -76,7 +76,7 @@ static int __init pasemi_register_i2c_devices(void) } info.irq = irq_of_parse_and_map(node, 0); - if (info.irq == NO_IRQ) + if (!info.irq) info.irq = -1; if (find_i2c_driver(node, &info) < 0) diff --git a/arch/powerpc/platforms/pasemi/msi.c b/arch/powerpc/platforms/pasemi/msi.c index d9af76342d99..d9cd510c8865 100644 --- a/arch/powerpc/platforms/pasemi/msi.c +++ b/arch/powerpc/platforms/pasemi/msi.c @@ -68,7 +68,7 @@ static void pasemi_msi_teardown_msi_irqs(struct pci_dev *pdev) pr_debug("pasemi_msi_teardown_msi_irqs, pdev %p\n", pdev); for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); @@ -109,7 +109,7 @@ static int pasemi_msi_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) } virq = irq_create_mapping(msi_mpic->irqhost, hwirq); - if (virq == NO_IRQ) { + if (!virq) { pr_debug("pasemi_msi: failed mapping hwirq 0x%x\n", hwirq); msi_bitmap_free_hwirqs(&msi_mpic->msi_bitmap, hwirq, diff --git a/arch/powerpc/platforms/pasemi/setup.c b/arch/powerpc/platforms/pasemi/setup.c index e86c1bd08f1f..3182400cf48f 100644 --- a/arch/powerpc/platforms/pasemi/setup.c +++ b/arch/powerpc/platforms/pasemi/setup.c @@ -59,7 +59,7 @@ struct mce_regs { static struct mce_regs mce_regs[MAX_MCE_REGS]; static int num_mce_regs; -static int nmi_virq = NO_IRQ; +static int nmi_virq = 0; static void __noreturn pas_restart(char *cmd) @@ -105,7 +105,7 @@ static void pas_take_timebase(void) arch_spin_unlock(&timebase_lock); } -struct smp_ops_t pas_smp_ops = { +static struct smp_ops_t pas_smp_ops = { .probe = smp_mpic_probe, .message_pass = smp_mpic_message_pass, .kick_cpu = smp_generic_kick_cpu, @@ -115,7 +115,7 @@ struct smp_ops_t pas_smp_ops = { }; #endif /* CONFIG_SMP */ -void __init pas_setup_arch(void) +static void __init pas_setup_arch(void) { #ifdef CONFIG_SMP /* Setup SMP callback */ @@ -264,7 +264,7 @@ static int pas_machine_check_handler(struct pt_regs *regs) srr0 = regs->nip; srr1 = regs->msr; - if (nmi_virq != NO_IRQ && mpic_get_mcirq() == nmi_virq) { + if (nmi_virq && mpic_get_mcirq() == nmi_virq) { printk(KERN_ERR "NMI delivered\n"); debugger(regs); mpic_end_irq(irq_get_irq_data(nmi_virq)); diff --git a/arch/powerpc/platforms/powermac/low_i2c.c b/arch/powerpc/platforms/powermac/low_i2c.c index 6d6f277477aa..c8c217b7dd33 100644 --- a/arch/powerpc/platforms/powermac/low_i2c.c +++ b/arch/powerpc/platforms/powermac/low_i2c.c @@ -401,7 +401,7 @@ static int kw_i2c_xfer(struct pmac_i2c_bus *bus, u8 addrdir, int subsize, { struct pmac_i2c_host_kw *host = bus->hostdata; u8 mode_reg = host->speed; - int use_irq = host->irq != NO_IRQ && !bus->polled; + int use_irq = host->irq && !bus->polled; /* Setup mode & subaddress if any */ switch(bus->mode) { @@ -535,7 +535,7 @@ static struct pmac_i2c_host_kw *__init kw_i2c_host_init(struct device_node *np) break; } host->irq = irq_of_parse_and_map(np, 0); - if (host->irq == NO_IRQ) + if (!host->irq) printk(KERN_WARNING "low_i2c: Failed to map interrupt for %s\n", np->full_name); @@ -557,7 +557,7 @@ static struct pmac_i2c_host_kw *__init kw_i2c_host_init(struct device_node *np) */ if (request_irq(host->irq, kw_i2c_irq, IRQF_NO_SUSPEND, "keywest i2c", host)) - host->irq = NO_IRQ; + host->irq = 0; printk(KERN_INFO "KeyWest i2c @0x%08x irq %d %s\n", *addrp, host->irq, np->full_name); diff --git a/arch/powerpc/platforms/powermac/pfunc_base.c b/arch/powerpc/platforms/powermac/pfunc_base.c index e49d07f3d542..459138ed4571 100644 --- a/arch/powerpc/platforms/powermac/pfunc_base.c +++ b/arch/powerpc/platforms/powermac/pfunc_base.c @@ -26,7 +26,7 @@ static irqreturn_t macio_gpio_irq(int irq, void *data) static int macio_do_gpio_irq_enable(struct pmf_function *func) { unsigned int irq = irq_of_parse_and_map(func->node, 0); - if (irq == NO_IRQ) + if (!irq) return -EINVAL; return request_irq(irq, macio_gpio_irq, 0, func->node->name, func); } @@ -34,7 +34,7 @@ static int macio_do_gpio_irq_enable(struct pmf_function *func) static int macio_do_gpio_irq_disable(struct pmf_function *func) { unsigned int irq = irq_of_parse_and_map(func->node, 0); - if (irq == NO_IRQ) + if (!irq) return -EINVAL; free_irq(irq, func); return 0; diff --git a/arch/powerpc/platforms/powermac/pfunc_core.c b/arch/powerpc/platforms/powermac/pfunc_core.c index 43075081721f..695e8c4d4224 100644 --- a/arch/powerpc/platforms/powermac/pfunc_core.c +++ b/arch/powerpc/platforms/powermac/pfunc_core.c @@ -804,7 +804,7 @@ void pmf_unregister_driver(struct device_node *np) } EXPORT_SYMBOL_GPL(pmf_unregister_driver); -struct pmf_function *__pmf_find_function(struct device_node *target, +static struct pmf_function *__pmf_find_function(struct device_node *target, const char *name, u32 flags) { struct device_node *actor = of_node_get(target); diff --git a/arch/powerpc/platforms/powermac/pic.c b/arch/powerpc/platforms/powermac/pic.c index 981546345033..f5f9ad7c3398 100644 --- a/arch/powerpc/platforms/powermac/pic.c +++ b/arch/powerpc/platforms/powermac/pic.c @@ -251,7 +251,7 @@ static unsigned int pmac_pic_get_irq(void) } raw_spin_unlock_irqrestore(&pmac_pic_lock, flags); if (unlikely(irq < 0)) - return NO_IRQ; + return 0; return irq_linear_revmap(pmac_pic_host, irq); } @@ -389,7 +389,7 @@ static void __init pmac_pic_probe_oldstyle(void) out_le32(&pmac_irq_hw[i]->enable, 0); /* Hookup cascade irq */ - if (slave && pmac_irq_cascade != NO_IRQ) + if (slave && pmac_irq_cascade) setup_irq(pmac_irq_cascade, &gatwick_cascade_action); printk(KERN_INFO "irq: System has %d possible interrupts\n", max_irqs); @@ -444,7 +444,7 @@ static void __init pmac_pic_setup_mpic_nmi(struct mpic *mpic) pswitch = of_find_node_by_name(NULL, "programmer-switch"); if (pswitch) { nmi_irq = irq_of_parse_and_map(pswitch, 0); - if (nmi_irq != NO_IRQ) { + if (nmi_irq) { mpic_irq_set_priority(nmi_irq, 9); setup_irq(nmi_irq, &xmon_action); } diff --git a/arch/powerpc/platforms/powermac/smp.c b/arch/powerpc/platforms/powermac/smp.c index 366e4f510fcf..c9eb7d6540ea 100644 --- a/arch/powerpc/platforms/powermac/smp.c +++ b/arch/powerpc/platforms/powermac/smp.c @@ -979,7 +979,7 @@ static void pmac_cpu_die(void) #endif /* CONFIG_HOTPLUG_CPU */ /* Core99 Macs (dual G4s and G5s) */ -struct smp_ops_t core99_smp_ops = { +static struct smp_ops_t core99_smp_ops = { .message_pass = smp_mpic_message_pass, .probe = smp_core99_probe, #ifdef CONFIG_PPC64 diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c index 86544ea85dc3..2354ea51e871 100644 --- a/arch/powerpc/platforms/powernv/eeh-powernv.c +++ b/arch/powerpc/platforms/powernv/eeh-powernv.c @@ -763,7 +763,8 @@ int pnv_eeh_phb_reset(struct pci_controller *hose, int option) * reset followed by hot reset on root bus. So we also * need the PCI bus settlement delay. */ - rc = pnv_eeh_poll(phb->opal_id); + if (rc > 0) + rc = pnv_eeh_poll(phb->opal_id); if (option == EEH_RESET_DEACTIVATE) { if (system_state < SYSTEM_RUNNING) udelay(1000 * EEH_PE_RST_SETTLE_TIME); @@ -806,7 +807,8 @@ static int pnv_eeh_root_reset(struct pci_controller *hose, int option) goto out; /* Poll state of the PHB until the request is done */ - rc = pnv_eeh_poll(phb->opal_id); + if (rc > 0) + rc = pnv_eeh_poll(phb->opal_id); if (option == EEH_RESET_DEACTIVATE) msleep(EEH_PE_RST_SETTLE_TIME); out: @@ -1090,10 +1092,16 @@ static int pnv_eeh_reset(struct eeh_pe *pe, int option) } } - bus = eeh_pe_bus_get(pe); if (pe->type & EEH_PE_VF) return pnv_eeh_reset_vf_pe(pe, option); + bus = eeh_pe_bus_get(pe); + if (!bus) { + pr_err("%s: Cannot find PCI bus for PHB#%d-PE#%x\n", + __func__, pe->phb->global_number, pe->addr); + return -EIO; + } + if (pci_is_root_bus(bus) || pci_is_root_bus(bus->parent)) return pnv_eeh_root_reset(hose, option); @@ -1306,7 +1314,7 @@ static void pnv_eeh_get_and_dump_hub_diag(struct pci_controller *hose) return; } - switch (data->type) { + switch (be16_to_cpu(data->type)) { case OPAL_P7IOC_DIAG_TYPE_RGC: pr_info("P7IOC diag-data for RGC\n\n"); pnv_eeh_dump_hub_diag_common(data); @@ -1538,7 +1546,7 @@ static int pnv_eeh_next_error(struct eeh_pe **pe) /* Try best to clear it */ opal_pci_eeh_freeze_clear(phb->opal_id, - frozen_pe_no, + be64_to_cpu(frozen_pe_no), OPAL_EEH_ACTION_CLEAR_FREEZE_ALL); ret = EEH_NEXT_ERR_NONE; } else if ((*pe)->state & EEH_PE_ISOLATED || diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c index 00e1a0195c78..aec85e778028 100644 --- a/arch/powerpc/platforms/powernv/npu-dma.c +++ b/arch/powerpc/platforms/powernv/npu-dma.c @@ -115,7 +115,7 @@ static u64 dma_npu_get_required_mask(struct device *dev) return 0; } -struct dma_map_ops dma_npu_ops = { +static struct dma_map_ops dma_npu_ops = { .map_page = dma_npu_map_page, .map_sg = dma_npu_map_sg, .alloc = dma_npu_alloc, diff --git a/arch/powerpc/platforms/powernv/opal-irqchip.c b/arch/powerpc/platforms/powernv/opal-irqchip.c index ed8bba68a162..998316bf2dad 100644 --- a/arch/powerpc/platforms/powernv/opal-irqchip.c +++ b/arch/powerpc/platforms/powernv/opal-irqchip.c @@ -222,7 +222,7 @@ int __init opal_event_init(void) /* Get hardware and virtual IRQ */ irq = be32_to_cpup(irqs); virq = irq_create_mapping(NULL, irq); - if (virq == NO_IRQ) { + if (!virq) { pr_warn("Failed to map irq 0x%x\n", irq); continue; } @@ -260,7 +260,7 @@ machine_arch_initcall(powernv, opal_event_init); int opal_event_request(unsigned int opal_event_nr) { if (WARN_ON_ONCE(!opal_event_irqchip.domain)) - return NO_IRQ; + return 0; return irq_create_mapping(opal_event_irqchip.domain, opal_event_nr); } diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c index 1349a099c74c..94498a04558b 100644 --- a/arch/powerpc/platforms/powernv/pci-cxl.c +++ b/arch/powerpc/platforms/powernv/pci-cxl.c @@ -344,7 +344,7 @@ int pnv_cxl_cx4_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) return (hwirq ? hwirq : -ENOMEM); virq = irq_create_mapping(NULL, hwirq); - if (virq == NO_IRQ) { + if (!virq) { pr_warn("%s: Failed to map cxl mode MSI to linux irq\n", pci_name(pdev)); return -ENOMEM; @@ -374,7 +374,7 @@ void pnv_cxl_cx4_teardown_msi_irqs(struct pci_dev *pdev) return; for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); irq_set_msi_desc(entry->irq, NULL); diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index d314eccd075b..d4b33dd2d9e7 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -133,9 +133,22 @@ static inline bool pnv_pci_is_m64_flags(unsigned long resource_flags) static struct pnv_ioda_pe *pnv_ioda_init_pe(struct pnv_phb *phb, int pe_no) { + s64 rc; + phb->ioda.pe_array[pe_no].phb = phb; phb->ioda.pe_array[pe_no].pe_number = pe_no; + /* + * Clear the PE frozen state as it might be put into frozen state + * in the last PCI remove path. It's not harmful to do so when the + * PE is already in unfrozen state. + */ + rc = opal_pci_eeh_freeze_clear(phb->opal_id, pe_no, + OPAL_EEH_ACTION_CLEAR_FREEZE_ALL); + if (rc != OPAL_SUCCESS) + pr_warn("%s: Error %lld unfreezing PHB#%d-PE#%d\n", + __func__, rc, phb->hose->global_number, pe_no); + return &phb->ioda.pe_array[pe_no]; } @@ -417,7 +430,7 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb) struct device_node *dn = hose->dn; struct resource *res; u32 m64_range[2], i; - const u32 *r; + const __be32 *r; u64 pci_addr; if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) { @@ -3049,6 +3062,38 @@ static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe) } } +#ifdef CONFIG_DEBUG_FS +static int pnv_pci_diag_data_set(void *data, u64 val) +{ + struct pci_controller *hose; + struct pnv_phb *phb; + s64 ret; + + if (val != 1ULL) + return -EINVAL; + + hose = (struct pci_controller *)data; + if (!hose || !hose->private_data) + return -ENODEV; + + phb = hose->private_data; + + /* Retrieve the diag data from firmware */ + ret = opal_pci_get_phb_diag_data2(phb->opal_id, phb->diag.blob, + PNV_PCI_DIAG_BUF_SIZE); + if (ret != OPAL_SUCCESS) + return -EIO; + + /* Print the diag data to the kernel log */ + pnv_pci_dump_phb_diag_data(phb->hose, phb->diag.blob); + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(pnv_pci_diag_data_fops, NULL, + pnv_pci_diag_data_set, "%llu\n"); + +#endif /* CONFIG_DEBUG_FS */ + static void pnv_pci_ioda_create_dbgfs(void) { #ifdef CONFIG_DEBUG_FS @@ -3064,9 +3109,14 @@ static void pnv_pci_ioda_create_dbgfs(void) sprintf(name, "PCI%04x", hose->global_number); phb->dbgfs = debugfs_create_dir(name, powerpc_debugfs_root); - if (!phb->dbgfs) + if (!phb->dbgfs) { pr_warning("%s: Error on creating debugfs on PHB#%x\n", __func__, hose->global_number); + continue; + } + + debugfs_create_file("dump_diag_regs", 0200, phb->dbgfs, hose, + &pnv_pci_diag_data_fops); } #endif /* CONFIG_DEBUG_FS */ } @@ -3779,10 +3829,11 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np, if (rc) pr_warning(" OPAL Error %ld performing IODA table reset !\n", rc); - /* If we're running in kdump kerenl, the previous kerenl never + /* + * If we're running in kdump kernel, the previous kernel never * shutdown PCI devices correctly. We already got IODA table * cleaned out. So we have to issue PHB reset to stop all PCI - * transactions from previous kerenl. + * transactions from previous kernel. */ if (is_kdump_kernel()) { pr_info(" Issue PHB reset ...\n"); diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c index a21d831c1114..db7b8020f68e 100644 --- a/arch/powerpc/platforms/powernv/pci.c +++ b/arch/powerpc/platforms/powernv/pci.c @@ -186,7 +186,7 @@ int pnv_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) return -ENOSPC; } virq = irq_create_mapping(NULL, phb->msi_base + hwirq); - if (virq == NO_IRQ) { + if (!virq) { pr_warn("%s: Failed to map MSI to linux irq\n", pci_name(pdev)); msi_bitmap_free_hwirqs(&phb->msi_bmp, hwirq, 1); @@ -217,7 +217,7 @@ void pnv_teardown_msi_irqs(struct pci_dev *pdev) return; for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); irq_set_msi_desc(entry->irq, NULL); @@ -309,8 +309,8 @@ static void pnv_pci_dump_p7ioc_diag_data(struct pci_controller *hose, be64_to_cpu(data->dma1ErrorLog1)); for (i = 0; i < OPAL_P7IOC_NUM_PEST_REGS; i++) { - if ((data->pestA[i] >> 63) == 0 && - (data->pestB[i] >> 63) == 0) + if ((be64_to_cpu(data->pestA[i]) >> 63) == 0 && + (be64_to_cpu(data->pestB[i]) >> 63) == 0) continue; pr_info("PE[%3d] A/B: %016llx %016llx\n", diff --git a/arch/powerpc/platforms/ps3/device-init.c b/arch/powerpc/platforms/ps3/device-init.c index 57caaf11a83f..e48462447ff0 100644 --- a/arch/powerpc/platforms/ps3/device-init.c +++ b/arch/powerpc/platforms/ps3/device-init.c @@ -62,7 +62,7 @@ static int __init ps3_register_lpm_devices(void) &dev->lpm.rights); if (result) { - pr_debug("%s:%d: ps3_repository_read_lpm_privleges failed \n", + pr_debug("%s:%d: ps3_repository_read_lpm_privileges failed\n", __func__, __LINE__); goto fail_read_repo; } diff --git a/arch/powerpc/platforms/ps3/interrupt.c b/arch/powerpc/platforms/ps3/interrupt.c index b831638e6f4a..98f8c3611133 100644 --- a/arch/powerpc/platforms/ps3/interrupt.c +++ b/arch/powerpc/platforms/ps3/interrupt.c @@ -192,7 +192,7 @@ static int ps3_virq_setup(enum ps3_cpu_binding cpu, unsigned long outlet, *virq = irq_create_mapping(NULL, outlet); - if (*virq == NO_IRQ) { + if (!*virq) { FAIL("%s:%d: irq_create_mapping failed: outlet %lu\n", __func__, __LINE__, outlet); result = -ENOMEM; @@ -339,7 +339,7 @@ int ps3_event_receive_port_setup(enum ps3_cpu_binding cpu, unsigned int *virq) if (result) { FAIL("%s:%d: lv1_construct_event_receive_port failed: %s\n", __func__, __LINE__, ps3_result(result)); - *virq = NO_IRQ; + *virq = 0; return result; } @@ -418,7 +418,7 @@ int ps3_sb_event_receive_port_setup(struct ps3_system_bus_device *dev, " failed: %s\n", __func__, __LINE__, ps3_result(result)); ps3_event_receive_port_destroy(*virq); - *virq = NO_IRQ; + *virq = 0; return result; } @@ -724,12 +724,12 @@ static unsigned int ps3_get_irq(void) asm volatile("cntlzd %0,%1" : "=r" (plug) : "r" (x)); plug &= 0x3f; - if (unlikely(plug == NO_IRQ)) { + if (unlikely(!plug)) { DBG("%s:%d: no plug found: thread_id %llu\n", __func__, __LINE__, pd->thread_id); dump_bmp(&per_cpu(ps3_private, 0)); dump_bmp(&per_cpu(ps3_private, 1)); - return NO_IRQ; + return 0; } #if defined(DEBUG) diff --git a/arch/powerpc/platforms/ps3/smp.c b/arch/powerpc/platforms/ps3/smp.c index 3c7707af3384..60154d08debf 100644 --- a/arch/powerpc/platforms/ps3/smp.c +++ b/arch/powerpc/platforms/ps3/smp.c @@ -91,7 +91,7 @@ static void __init ps3_smp_probe(void) result = smp_request_message_ipi(virqs[i], i); if (result) - virqs[i] = NO_IRQ; + virqs[i] = 0; else ps3_register_ipi_irq(cpu, virqs[i]); } @@ -112,7 +112,7 @@ void ps3_smp_cleanup_cpu(int cpu) for (i = 0; i < MSG_COUNT; i++) { /* Can't call free_irq from interrupt context. */ ps3_event_receive_port_destroy(virqs[i]); - virqs[i] = NO_IRQ; + virqs[i] = 0; } DBG(" <- %s:%d: (%d)\n", __func__, __LINE__, cpu); diff --git a/arch/powerpc/platforms/ps3/spu.c b/arch/powerpc/platforms/ps3/spu.c index 492b2575e0d2..b54850845466 100644 --- a/arch/powerpc/platforms/ps3/spu.c +++ b/arch/powerpc/platforms/ps3/spu.c @@ -284,7 +284,7 @@ fail_alloc_2: fail_alloc_1: ps3_spe_irq_destroy(spu->irqs[0]); fail_alloc_0: - spu->irqs[0] = spu->irqs[1] = spu->irqs[2] = NO_IRQ; + spu->irqs[0] = spu->irqs[1] = spu->irqs[2] = 0; return result; } @@ -334,7 +334,7 @@ static int ps3_destroy_spu(struct spu *spu) ps3_spe_irq_destroy(spu->irqs[1]); ps3_spe_irq_destroy(spu->irqs[0]); - spu->irqs[0] = spu->irqs[1] = spu->irqs[2] = NO_IRQ; + spu->irqs[0] = spu->irqs[1] = spu->irqs[2] = 0; spu_unmap(spu); diff --git a/arch/powerpc/platforms/pseries/dlpar.c b/arch/powerpc/platforms/pseries/dlpar.c index 4748124faa10..423e450efe07 100644 --- a/arch/powerpc/platforms/pseries/dlpar.c +++ b/arch/powerpc/platforms/pseries/dlpar.c @@ -27,7 +27,7 @@ #include <asm/uaccess.h> #include <asm/rtas.h> -struct workqueue_struct *pseries_hp_wq; +static struct workqueue_struct *pseries_hp_wq; struct pseries_hp_work { struct work_struct work; @@ -377,7 +377,7 @@ static int handle_dlpar_errorlog(struct pseries_hp_errorlog *hp_elog) return rc; } -void pseries_hp_work_fn(struct work_struct *work) +static void pseries_hp_work_fn(struct work_struct *work) { struct pseries_hp_work *hp_work = container_of(work, struct pseries_hp_work, work); @@ -413,6 +413,7 @@ void queue_hotplug_event(struct pseries_hp_errorlog *hp_errlog, queue_work(pseries_hp_wq, (struct work_struct *)work); } else { *rc = -ENOMEM; + kfree(hp_errlog_copy); complete(hotplug_done); } } diff --git a/arch/powerpc/platforms/pseries/event_sources.c b/arch/powerpc/platforms/pseries/event_sources.c index a6ddca833119..32187dc76730 100644 --- a/arch/powerpc/platforms/pseries/event_sources.c +++ b/arch/powerpc/platforms/pseries/event_sources.c @@ -34,7 +34,7 @@ void request_event_sources_irqs(struct device_node *np, if (count > 15) break; virqs[count] = irq_create_of_mapping(&oirq); - if (virqs[count] == NO_IRQ) { + if (!virqs[count]) { pr_err("event-sources: Unable to allocate " "interrupt number for %s\n", np->full_name); diff --git a/arch/powerpc/platforms/pseries/msi.c b/arch/powerpc/platforms/pseries/msi.c index 543a6386f3eb..326ef0dd6038 100644 --- a/arch/powerpc/platforms/pseries/msi.c +++ b/arch/powerpc/platforms/pseries/msi.c @@ -119,7 +119,7 @@ static void rtas_teardown_msi_irqs(struct pci_dev *pdev) struct msi_desc *entry; for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; irq_set_msi_desc(entry->irq, NULL); @@ -471,7 +471,7 @@ again: virq = irq_create_mapping(NULL, hwirq); - if (virq == NO_IRQ) { + if (!virq) { pr_debug("rtas_msi: Failed mapping hwirq %d\n", hwirq); return -ENOSPC; } @@ -490,7 +490,7 @@ again: static void rtas_msi_pci_irq_fixup(struct pci_dev *pdev) { /* No LSI -> leave MSIs (if any) configured */ - if (pdev->irq == NO_IRQ) { + if (!pdev->irq) { dev_dbg(&pdev->dev, "rtas_msi: no LSI, nothing to do.\n"); return; } diff --git a/arch/powerpc/platforms/pseries/scanlog.c b/arch/powerpc/platforms/pseries/scanlog.c index b502ab61aafa..7d28cabf1206 100644 --- a/arch/powerpc/platforms/pseries/scanlog.c +++ b/arch/powerpc/platforms/pseries/scanlog.c @@ -156,7 +156,7 @@ static int scanlog_release(struct inode * inode, struct file * file) return 0; } -const struct file_operations scanlog_fops = { +static const struct file_operations scanlog_fops = { .owner = THIS_MODULE, .read = scanlog_read, .write = scanlog_write, diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index a39d20e8623d..97aa3f332f24 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -114,7 +114,7 @@ static void pseries_8259_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = i8259_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); @@ -141,7 +141,7 @@ static void __init pseries_setup_i8259_cascade(void) } cascade = irq_of_parse_and_map(found, 0); - if (cascade == NO_IRQ) { + if (!cascade) { printk(KERN_ERR "pic: failed to map cascade interrupt"); return; } diff --git a/arch/powerpc/sysdev/axonram.c b/arch/powerpc/sysdev/axonram.c index 9144204442eb..ada29eaed6e2 100644 --- a/arch/powerpc/sysdev/axonram.c +++ b/arch/powerpc/sysdev/axonram.c @@ -240,7 +240,7 @@ static int axon_ram_probe(struct platform_device *device) device_add_disk(&device->dev, bank->disk); bank->irq_id = irq_of_parse_and_map(device->dev.of_node, 0); - if (bank->irq_id == NO_IRQ) { + if (!bank->irq_id) { dev_err(&device->dev, "Cannot access ECC interrupt ID\n"); rc = -EFAULT; goto failed; @@ -250,7 +250,7 @@ static int axon_ram_probe(struct platform_device *device) AXON_RAM_IRQ_FLAGS, bank->disk->disk_name, device); if (rc != 0) { dev_err(&device->dev, "Cannot register ECC interrupt handler\n"); - bank->irq_id = NO_IRQ; + bank->irq_id = 0; rc = -EFAULT; goto failed; } @@ -268,7 +268,7 @@ static int axon_ram_probe(struct platform_device *device) failed: if (bank != NULL) { - if (bank->irq_id != NO_IRQ) + if (bank->irq_id) free_irq(bank->irq_id, device); if (bank->disk != NULL) { if (bank->disk->major > 0) diff --git a/arch/powerpc/sysdev/cpm1.c b/arch/powerpc/sysdev/cpm1.c index 81d49476c47e..3c0eb9b25535 100644 --- a/arch/powerpc/sysdev/cpm1.c +++ b/arch/powerpc/sysdev/cpm1.c @@ -132,7 +132,7 @@ unsigned int cpm_pic_init(void) { struct device_node *np = NULL; struct resource res; - unsigned int sirq = NO_IRQ, hwirq, eirq; + unsigned int sirq = 0, hwirq, eirq; int ret; pr_debug("cpm_pic_init\n"); @@ -154,7 +154,7 @@ unsigned int cpm_pic_init(void) goto end; sirq = irq_of_parse_and_map(np, 0); - if (sirq == NO_IRQ) + if (!sirq) goto end; /* Initialize the CPM interrupt controller. */ @@ -168,7 +168,7 @@ unsigned int cpm_pic_init(void) cpm_pic_host = irq_domain_add_linear(np, 64, &cpm_pic_host_ops, NULL); if (cpm_pic_host == NULL) { printk(KERN_ERR "CPM2 PIC: failed to allocate irq host!\n"); - sirq = NO_IRQ; + sirq = 0; goto end; } @@ -182,7 +182,7 @@ unsigned int cpm_pic_init(void) } eirq = irq_of_parse_and_map(np, 0); - if (eirq == NO_IRQ) + if (!eirq) goto end; if (setup_irq(eirq, &cpm_error_irqaction)) diff --git a/arch/powerpc/sysdev/ehv_pic.c b/arch/powerpc/sysdev/ehv_pic.c index bffcc7a486a1..48866e6c1efb 100644 --- a/arch/powerpc/sysdev/ehv_pic.c +++ b/arch/powerpc/sysdev/ehv_pic.c @@ -155,7 +155,7 @@ static struct irq_chip ehv_pic_direct_eoi_irq_chip = { .irq_set_type = ehv_pic_set_irq_type, }; -/* Return an interrupt vector or NO_IRQ if no interrupt is pending. */ +/* Return an interrupt vector or 0 if no interrupt is pending. */ unsigned int ehv_pic_get_irq(void) { int irq; @@ -168,7 +168,7 @@ unsigned int ehv_pic_get_irq(void) ev_int_iack(0, &irq); /* legacy mode */ if (irq == 0xFFFF) /* 0xFFFF --> no irq is pending */ - return NO_IRQ; + return 0; /* * this will also setup revmap[] in the slow path for the first diff --git a/arch/powerpc/sysdev/fsl_gtm.c b/arch/powerpc/sysdev/fsl_gtm.c index 06ac3c61b3d0..a6f0b96ce2c9 100644 --- a/arch/powerpc/sysdev/fsl_gtm.c +++ b/arch/powerpc/sysdev/fsl_gtm.c @@ -406,7 +406,7 @@ static int __init fsl_gtm_init(void) unsigned int irq; irq = irq_of_parse_and_map(np, i); - if (irq == NO_IRQ) { + if (!irq) { pr_err("%s: not enough interrupts specified\n", np->full_name); goto err; diff --git a/arch/powerpc/sysdev/fsl_mpic_err.c b/arch/powerpc/sysdev/fsl_mpic_err.c index b83f32562a37..488ec453038a 100644 --- a/arch/powerpc/sysdev/fsl_mpic_err.c +++ b/arch/powerpc/sysdev/fsl_mpic_err.c @@ -115,8 +115,8 @@ static irqreturn_t fsl_error_int_handler(int irq, void *data) errint = __builtin_clz(eisr); cascade_irq = irq_linear_revmap(mpic->irqhost, mpic->err_int_vecs[errint]); - WARN_ON(cascade_irq == NO_IRQ); - if (cascade_irq != NO_IRQ) { + WARN_ON(!cascade_irq); + if (cascade_irq) { generic_handle_irq(cascade_irq); } else { eimr |= 1 << (31 - errint); @@ -134,7 +134,7 @@ void mpic_err_int_init(struct mpic *mpic, irq_hw_number_t irqnum) int ret; virq = irq_create_mapping(mpic->irqhost, irqnum); - if (virq == NO_IRQ) { + if (!virq) { pr_err("Error interrupt setup failed\n"); return; } diff --git a/arch/powerpc/sysdev/fsl_msi.c b/arch/powerpc/sysdev/fsl_msi.c index 3a2be3676f43..8a244828782e 100644 --- a/arch/powerpc/sysdev/fsl_msi.c +++ b/arch/powerpc/sysdev/fsl_msi.c @@ -131,7 +131,7 @@ static void fsl_teardown_msi_irqs(struct pci_dev *pdev) irq_hw_number_t hwirq; for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); msi_data = irq_get_chip_data(entry->irq); @@ -250,7 +250,7 @@ static int fsl_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) virq = irq_create_mapping(msi_data->irqhost, hwirq); - if (virq == NO_IRQ) { + if (!virq) { dev_err(&pdev->dev, "fail mapping hwirq %i\n", hwirq); msi_bitmap_free_hwirqs(&msi_data->bitmap, hwirq, 1); rc = -ENOSPC; @@ -285,7 +285,7 @@ static irqreturn_t fsl_msi_cascade(int irq, void *data) msir_index = cascade_data->index; if (msir_index >= NR_MSI_REG_MAX) - cascade_irq = NO_IRQ; + cascade_irq = 0; switch (msi_data->feature & FSL_PIC_IP_MASK) { case FSL_PIC_IP_MPIC: @@ -315,7 +315,7 @@ static irqreturn_t fsl_msi_cascade(int irq, void *data) cascade_irq = irq_linear_revmap(msi_data->irqhost, msi_hwirq(msi_data, msir_index, intr_index + have_shift)); - if (cascade_irq != NO_IRQ) { + if (cascade_irq) { generic_handle_irq(cascade_irq); ret = IRQ_HANDLED; } @@ -337,7 +337,7 @@ static int fsl_of_msi_remove(struct platform_device *ofdev) if (msi->cascade_array[i]) { virq = msi->cascade_array[i]->virq; - BUG_ON(virq == NO_IRQ); + BUG_ON(!virq); free_irq(virq, msi->cascade_array[i]); kfree(msi->cascade_array[i]); @@ -362,7 +362,7 @@ static int fsl_msi_setup_hwirq(struct fsl_msi *msi, struct platform_device *dev, int virt_msir, i, ret; virt_msir = irq_of_parse_and_map(dev->dev.of_node, irq_index); - if (virt_msir == NO_IRQ) { + if (!virt_msir) { dev_err(&dev->dev, "%s: Cannot translate IRQ index %d\n", __func__, irq_index); return 0; diff --git a/arch/powerpc/sysdev/ge/ge_pic.c b/arch/powerpc/sysdev/ge/ge_pic.c index d57b77573068..02553a8ce191 100644 --- a/arch/powerpc/sysdev/ge/ge_pic.c +++ b/arch/powerpc/sysdev/ge/ge_pic.c @@ -102,7 +102,7 @@ static void gef_pic_cascade(struct irq_desc *desc) */ cascade_irq = gef_pic_get_irq(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); @@ -206,7 +206,7 @@ void __init gef_pic_init(struct device_node *np) /* Map controller */ gef_pic_cascade_irq = irq_of_parse_and_map(np, 0); - if (gef_pic_cascade_irq == NO_IRQ) { + if (!gef_pic_cascade_irq) { printk(KERN_ERR "SBC610: failed to map cascade interrupt"); return; } @@ -223,12 +223,12 @@ void __init gef_pic_init(struct device_node *np) /* * This is called when we receive an interrupt with apparently comes from this - * chip - check, returning the highest interrupt generated or return NO_IRQ + * chip - check, returning the highest interrupt generated or return 0. */ unsigned int gef_pic_get_irq(void) { u32 cause, mask, active; - unsigned int virq = NO_IRQ; + unsigned int virq = 0; int hwirq; cause = in_be32(gef_pic_irq_reg_base + GEF_PIC_INTR_STATUS); diff --git a/arch/powerpc/sysdev/i8259.c b/arch/powerpc/sysdev/i8259.c index aa2c186d3115..bafb014e1a7e 100644 --- a/arch/powerpc/sysdev/i8259.c +++ b/arch/powerpc/sysdev/i8259.c @@ -68,9 +68,9 @@ unsigned int i8259_irq(void) if (!pci_intack) outb(0x0B, 0x20); /* ISR register */ if(~inb(0x20) & 0x80) - irq = NO_IRQ; + irq = 0; } else if (irq == 0xff) - irq = NO_IRQ; + irq = 0; if (lock) raw_spin_unlock(&i8259_lock); diff --git a/arch/powerpc/sysdev/ipic.c b/arch/powerpc/sysdev/ipic.c index f76ee39cb337..f267ee0afc08 100644 --- a/arch/powerpc/sysdev/ipic.c +++ b/arch/powerpc/sysdev/ipic.c @@ -853,7 +853,7 @@ void ipic_clear_mcp_status(u32 mask) ipic_write(primary_ipic->regs, IPIC_SERMR, mask); } -/* Return an interrupt vector or NO_IRQ if no interrupt is pending. */ +/* Return an interrupt vector or 0 if no interrupt is pending. */ unsigned int ipic_get_irq(void) { int irq; @@ -864,7 +864,7 @@ unsigned int ipic_get_irq(void) irq = ipic_read(primary_ipic->regs, IPIC_SIVCR) & IPIC_SIVCR_VECTOR_MASK; if (irq == 0) /* 0 --> no irq is pending */ - return NO_IRQ; + return 0; return irq_linear_revmap(primary_ipic->irqhost, irq); } diff --git a/arch/powerpc/sysdev/mmio_nvram.c b/arch/powerpc/sysdev/mmio_nvram.c index 69f5814ae6d4..c31f634f1973 100644 --- a/arch/powerpc/sysdev/mmio_nvram.c +++ b/arch/powerpc/sysdev/mmio_nvram.c @@ -89,7 +89,7 @@ static ssize_t mmio_nvram_write(char *buf, size_t count, loff_t *index) return count; } -void mmio_nvram_write_val(int addr, unsigned char val) +static void mmio_nvram_write_val(int addr, unsigned char val) { unsigned long flags; diff --git a/arch/powerpc/sysdev/mpc8xx_pic.c b/arch/powerpc/sysdev/mpc8xx_pic.c index b7cf7abff2eb..3e828b20c21e 100644 --- a/arch/powerpc/sysdev/mpc8xx_pic.c +++ b/arch/powerpc/sysdev/mpc8xx_pic.c @@ -79,7 +79,7 @@ unsigned int mpc8xx_get_irq(void) irq = in_be32(&siu_reg->sc_sivec) >> 26; if (irq == PIC_VEC_SPURRIOUS) - irq = NO_IRQ; + irq = 0; return irq_linear_revmap(mpc8xx_pic_host, irq); diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c index 7de45b2df366..4d48cecfedd1 100644 --- a/arch/powerpc/sysdev/mpic.c +++ b/arch/powerpc/sysdev/mpic.c @@ -1649,7 +1649,7 @@ void __init mpic_init(struct mpic *mpic) /* Check if this MPIC is chained from a parent interrupt controller */ if (mpic->flags & MPIC_SECONDARY) { int virq = irq_of_parse_and_map(mpic->node, 0); - if (virq != NO_IRQ) { + if (virq) { printk(KERN_INFO "%s: hooking up to IRQ %d\n", mpic->node->full_name, virq); irq_set_handler_data(virq, mpic); @@ -1778,13 +1778,13 @@ static unsigned int _mpic_get_one_irq(struct mpic *mpic, int reg) if (unlikely(src == mpic->spurious_vec)) { if (mpic->flags & MPIC_SPV_EOI) mpic_eoi(mpic); - return NO_IRQ; + return 0; } if (unlikely(mpic->protected && test_bit(src, mpic->protected))) { printk_ratelimited(KERN_WARNING "%s: Got protected source %d !\n", mpic->name, (int)src); mpic_eoi(mpic); - return NO_IRQ; + return 0; } return irq_linear_revmap(mpic->irqhost, src); @@ -1817,17 +1817,17 @@ unsigned int mpic_get_coreint_irq(void) if (unlikely(src == mpic->spurious_vec)) { if (mpic->flags & MPIC_SPV_EOI) mpic_eoi(mpic); - return NO_IRQ; + return 0; } if (unlikely(mpic->protected && test_bit(src, mpic->protected))) { printk_ratelimited(KERN_WARNING "%s: Got protected source %d !\n", mpic->name, (int)src); - return NO_IRQ; + return 0; } return irq_linear_revmap(mpic->irqhost, src); #else - return NO_IRQ; + return 0; #endif } @@ -1852,7 +1852,7 @@ void mpic_request_ipis(void) for (i = 0; i < 4; i++) { unsigned int vipi = irq_create_mapping(mpic->irqhost, mpic->ipi_vecs[0] + i); - if (vipi == NO_IRQ) { + if (!vipi) { printk(KERN_ERR "Failed to map %s\n", smp_ipi_name[i]); continue; } diff --git a/arch/powerpc/sysdev/mpic_msgr.c b/arch/powerpc/sysdev/mpic_msgr.c index 3f165d972a0e..db2286be5d9a 100644 --- a/arch/powerpc/sysdev/mpic_msgr.c +++ b/arch/powerpc/sysdev/mpic_msgr.c @@ -238,7 +238,7 @@ static int mpic_msgr_probe(struct platform_device *dev) if (receive_mask & (1 << i)) { msgr->irq = irq_of_parse_and_map(np, irq_index); - if (msgr->irq == NO_IRQ) { + if (!msgr->irq) { dev_err(&dev->dev, "Missing interrupt specifier"); kfree(msgr); @@ -246,7 +246,7 @@ static int mpic_msgr_probe(struct platform_device *dev) } irq_index += 1; } else { - msgr->irq = NO_IRQ; + msgr->irq = 0; } mpic_msgrs[reg_number] = msgr; diff --git a/arch/powerpc/sysdev/mpic_u3msi.c b/arch/powerpc/sysdev/mpic_u3msi.c index 2cbc7e29b85f..cfc1c57d760f 100644 --- a/arch/powerpc/sysdev/mpic_u3msi.c +++ b/arch/powerpc/sysdev/mpic_u3msi.c @@ -110,7 +110,7 @@ static void u3msi_teardown_msi_irqs(struct pci_dev *pdev) irq_hw_number_t hwirq; for_each_pci_msi_entry(entry, pdev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); @@ -155,7 +155,7 @@ static int u3msi_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type) msg.address_hi = addr >> 32; virq = irq_create_mapping(msi_mpic->irqhost, hwirq); - if (virq == NO_IRQ) { + if (!virq) { pr_debug("u3msi: failed mapping hwirq 0x%x\n", hwirq); msi_bitmap_free_hwirqs(&msi_mpic->msi_bitmap, hwirq, 1); return -ENOSPC; diff --git a/arch/powerpc/sysdev/mv64x60_pic.c b/arch/powerpc/sysdev/mv64x60_pic.c index 0f842dd16bcd..a79953deb489 100644 --- a/arch/powerpc/sysdev/mv64x60_pic.c +++ b/arch/powerpc/sysdev/mv64x60_pic.c @@ -272,7 +272,7 @@ unsigned int mv64x60_get_irq(void) u32 cause; int level1; irq_hw_number_t hwirq; - int virq = NO_IRQ; + int virq = 0; cause = in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_SELECT_CAUSE); if (cause & MV64X60_SELECT_CAUSE_HIGH) { diff --git a/arch/powerpc/sysdev/pmi.c b/arch/powerpc/sysdev/pmi.c index 8a0b77a3ec0c..9ea6a221d9d5 100644 --- a/arch/powerpc/sysdev/pmi.c +++ b/arch/powerpc/sysdev/pmi.c @@ -158,7 +158,7 @@ static int pmi_of_probe(struct platform_device *dev) data->dev = dev; data->irq = irq_of_parse_and_map(np, 0); - if (data->irq == NO_IRQ) { + if (!data->irq) { printk(KERN_ERR "pmi: invalid interrupt.\n"); rc = -EFAULT; goto error_cleanup_iomap; diff --git a/arch/powerpc/sysdev/ppc4xx_hsta_msi.c b/arch/powerpc/sysdev/ppc4xx_hsta_msi.c index 52a93dcae262..9926ad67af76 100644 --- a/arch/powerpc/sysdev/ppc4xx_hsta_msi.c +++ b/arch/powerpc/sysdev/ppc4xx_hsta_msi.c @@ -60,7 +60,7 @@ static int hsta_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) } hwirq = ppc4xx_hsta_msi.irq_map[irq]; - if (hwirq == NO_IRQ) { + if (!hwirq) { pr_err("%s: Failed mapping irq %d\n", __func__, irq); return -EINVAL; } @@ -110,7 +110,7 @@ static void hsta_teardown_msi_irqs(struct pci_dev *dev) int irq; for_each_pci_msi_entry(entry, dev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; irq = hsta_find_hwirq_offset(entry->irq); @@ -166,7 +166,7 @@ static int hsta_msi_probe(struct platform_device *pdev) for (irq = 0; irq < irq_count; irq++) { ppc4xx_hsta_msi.irq_map[irq] = irq_of_parse_and_map(dev->of_node, irq); - if (ppc4xx_hsta_msi.irq_map[irq] == NO_IRQ) { + if (!ppc4xx_hsta_msi.irq_map[irq]) { dev_err(dev, "Unable to map IRQ\n"); ret = -EINVAL; goto out2; diff --git a/arch/powerpc/sysdev/ppc4xx_msi.c b/arch/powerpc/sysdev/ppc4xx_msi.c index 8fb806135043..590dab4f47d6 100644 --- a/arch/powerpc/sysdev/ppc4xx_msi.c +++ b/arch/powerpc/sysdev/ppc4xx_msi.c @@ -102,7 +102,7 @@ static int ppc4xx_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) __func__); } virq = irq_of_parse_and_map(msi_data->msi_dev, int_no); - if (virq == NO_IRQ) { + if (!virq) { dev_err(&dev->dev, "%s: fail mapping irq\n", __func__); msi_bitmap_free_hwirqs(&msi_data->bitmap, int_no, 1); return -ENOSPC; @@ -129,7 +129,7 @@ void ppc4xx_teardown_msi_irqs(struct pci_dev *dev) dev_dbg(&dev->dev, "PCIE-MSI: tearing down msi irqs\n"); for_each_pci_msi_entry(entry, dev) { - if (entry->irq == NO_IRQ) + if (!entry->irq) continue; hwirq = virq_to_hw(entry->irq); irq_set_msi_desc(entry->irq, NULL); @@ -201,7 +201,7 @@ static int ppc4xx_of_msi_remove(struct platform_device *dev) for (i = 0; i < msi_irqs; i++) { virq = msi->msi_virqs[i]; - if (virq != NO_IRQ) + if (virq) irq_dispose_mapping(virq); } diff --git a/arch/powerpc/sysdev/ppc4xx_soc.c b/arch/powerpc/sysdev/ppc4xx_soc.c index 5c77c9ba33aa..d41134d2f786 100644 --- a/arch/powerpc/sysdev/ppc4xx_soc.c +++ b/arch/powerpc/sysdev/ppc4xx_soc.c @@ -109,7 +109,7 @@ static int __init ppc4xx_l2c_probe(void) /* Get and map irq number from device tree */ irq = irq_of_parse_and_map(np, 0); - if (irq == NO_IRQ) { + if (!irq) { printk(KERN_ERR "irq_of_parse_and_map failed\n"); of_node_put(np); return -ENODEV; diff --git a/arch/powerpc/sysdev/tsi108_pci.c b/arch/powerpc/sysdev/tsi108_pci.c index 379de955aae3..57c971b7839c 100644 --- a/arch/powerpc/sysdev/tsi108_pci.c +++ b/arch/powerpc/sysdev/tsi108_pci.c @@ -433,7 +433,7 @@ void tsi108_irq_cascade(struct irq_desc *desc) struct irq_chip *chip = irq_desc_get_chip(desc); unsigned int cascade_irq = get_pci_source(); - if (cascade_irq != NO_IRQ) + if (cascade_irq) generic_handle_irq(cascade_irq); chip->irq_eoi(&desc->irq_data); diff --git a/arch/powerpc/sysdev/uic.c b/arch/powerpc/sysdev/uic.c index 6893d8f236df..a00949f3e378 100644 --- a/arch/powerpc/sysdev/uic.c +++ b/arch/powerpc/sysdev/uic.c @@ -319,7 +319,7 @@ void __init uic_init_tree(void) } } -/* Return an interrupt vector or NO_IRQ if no interrupt is pending. */ +/* Return an interrupt vector or 0 if no interrupt is pending. */ unsigned int uic_get_irq(void) { u32 msr; diff --git a/arch/powerpc/sysdev/xics/icp-hv.c b/arch/powerpc/sysdev/xics/icp-hv.c index c1917cf67c3d..e7fa26c4ff73 100644 --- a/arch/powerpc/sysdev/xics/icp-hv.c +++ b/arch/powerpc/sysdev/xics/icp-hv.c @@ -112,10 +112,10 @@ static unsigned int icp_hv_get_irq(void) unsigned int irq; if (vec == XICS_IRQ_SPURIOUS) - return NO_IRQ; + return 0; irq = irq_find_mapping(xics_host, vec); - if (likely(irq != NO_IRQ)) { + if (likely(irq)) { xics_push_cppr(vec); return irq; } @@ -126,7 +126,7 @@ static unsigned int icp_hv_get_irq(void) /* We might learn about it later, so EOI it */ icp_hv_set_xirr(xirr); - return NO_IRQ; + return 0; } static void icp_hv_set_cpu_priority(unsigned char cppr) diff --git a/arch/powerpc/sysdev/xics/icp-native.c b/arch/powerpc/sysdev/xics/icp-native.c index afdf62f2a695..8a6a043e239b 100644 --- a/arch/powerpc/sysdev/xics/icp-native.c +++ b/arch/powerpc/sysdev/xics/icp-native.c @@ -124,10 +124,10 @@ static unsigned int icp_native_get_irq(void) unsigned int irq; if (vec == XICS_IRQ_SPURIOUS) - return NO_IRQ; + return 0; irq = irq_find_mapping(xics_host, vec); - if (likely(irq != NO_IRQ)) { + if (likely(irq)) { xics_push_cppr(vec); return irq; } @@ -138,7 +138,7 @@ static unsigned int icp_native_get_irq(void) /* We might learn about it later, so EOI it */ icp_native_set_xirr(xirr); - return NO_IRQ; + return 0; } #ifdef CONFIG_SMP diff --git a/arch/powerpc/sysdev/xics/icp-opal.c b/arch/powerpc/sysdev/xics/icp-opal.c index 9114243fa1b5..d38e86fd5720 100644 --- a/arch/powerpc/sysdev/xics/icp-opal.c +++ b/arch/powerpc/sysdev/xics/icp-opal.c @@ -51,14 +51,14 @@ static unsigned int icp_opal_get_irq(void) rc = opal_int_get_xirr(&xirr, false); if (rc < 0) - return NO_IRQ; + return 0; xirr = be32_to_cpu(xirr); vec = xirr & 0x00ffffff; if (vec == XICS_IRQ_SPURIOUS) - return NO_IRQ; + return 0; irq = irq_find_mapping(xics_host, vec); - if (likely(irq != NO_IRQ)) { + if (likely(irq)) { xics_push_cppr(vec); return irq; } @@ -69,7 +69,7 @@ static unsigned int icp_opal_get_irq(void) /* We might learn about it later, so EOI it */ opal_int_eoi(xirr); - return NO_IRQ; + return 0; } static void icp_opal_set_cpu_priority(unsigned char cppr) diff --git a/arch/powerpc/sysdev/xics/xics-common.c b/arch/powerpc/sysdev/xics/xics-common.c index 9d530f479588..69d858e51ac7 100644 --- a/arch/powerpc/sysdev/xics/xics-common.c +++ b/arch/powerpc/sysdev/xics/xics-common.c @@ -131,7 +131,7 @@ static void xics_request_ipi(void) unsigned int ipi; ipi = irq_create_mapping(xics_host, XICS_IPI); - BUG_ON(ipi == NO_IRQ); + BUG_ON(!ipi); /* * IPIs are marked IRQF_PERCPU. The handler was set in map. diff --git a/arch/powerpc/xmon/spr_access.S b/arch/powerpc/xmon/spr_access.S index 84ad74213c83..7d8b0e8ed6d9 100644 --- a/arch/powerpc/xmon/spr_access.S +++ b/arch/powerpc/xmon/spr_access.S @@ -2,12 +2,12 @@ /* unsigned long xmon_mfspr(sprn, default_value) */ _GLOBAL(xmon_mfspr) - ld r5, .Lmfspr_table@got(r2) + PPC_LL r5, .Lmfspr_table@got(r2) b xmon_mxspr /* void xmon_mtspr(sprn, new_value) */ _GLOBAL(xmon_mtspr) - ld r5, .Lmtspr_table@got(r2) + PPC_LL r5, .Lmtspr_table@got(r2) b xmon_mxspr /* diff --git a/drivers/char/hw_random/pasemi-rng.c b/drivers/char/hw_random/pasemi-rng.c index 699b7259f5d7..c19e23d22b36 100644 --- a/drivers/char/hw_random/pasemi-rng.c +++ b/drivers/char/hw_random/pasemi-rng.c @@ -26,7 +26,7 @@ #include <linux/delay.h> #include <linux/of_address.h> #include <linux/of_platform.h> -#include <asm/io.h> +#include <linux/io.h> #define SDCRNG_CTL_REG 0x00 #define SDCRNG_CTL_FVLD_M 0x0000f000 diff --git a/drivers/macintosh/macio_asic.c b/drivers/macintosh/macio_asic.c index b6819f0fc608..3f041b187033 100644 --- a/drivers/macintosh/macio_asic.c +++ b/drivers/macintosh/macio_asic.c @@ -236,7 +236,7 @@ static void macio_create_fixup_irq(struct macio_dev *dev, int index, unsigned int irq; irq = irq_create_mapping(NULL, line); - if (irq != NO_IRQ) { + if (!irq) { dev->interrupt[index].start = irq; dev->interrupt[index].flags = IORESOURCE_IRQ; dev->interrupt[index].name = dev_name(&dev->ofdev.dev); @@ -299,7 +299,7 @@ static void macio_setup_interrupts(struct macio_dev *dev) break; res = &dev->interrupt[j]; irq = irq_of_parse_and_map(np, i++); - if (irq == NO_IRQ) + if (!irq) break; res->start = irq; res->flags = IORESOURCE_IRQ; diff --git a/drivers/macintosh/rack-meter.c b/drivers/macintosh/rack-meter.c index 465c52219639..775527135b93 100644 --- a/drivers/macintosh/rack-meter.c +++ b/drivers/macintosh/rack-meter.c @@ -427,7 +427,7 @@ static int rackmeter_probe(struct macio_dev* mdev, rm->irq = macio_irq(mdev, 1); #else rm->irq = irq_of_parse_and_map(i2s, 1); - if (rm->irq == NO_IRQ || + if (!rm->irq || of_address_to_resource(i2s, 0, &ri2s) || of_address_to_resource(i2s, 1, &rdma)) { printk(KERN_ERR diff --git a/drivers/macintosh/smu.c b/drivers/macintosh/smu.c index d6f72c826c1c..08edb2c25b60 100644 --- a/drivers/macintosh/smu.c +++ b/drivers/macintosh/smu.c @@ -279,7 +279,7 @@ int smu_queue_cmd(struct smu_cmd *cmd) spin_unlock_irqrestore(&smu->lock, flags); /* Workaround for early calls when irq isn't available */ - if (!smu_irq_inited || smu->db_irq == NO_IRQ) + if (!smu_irq_inited || !smu->db_irq) smu_spinwait_cmd(cmd); return 0; @@ -498,8 +498,8 @@ int __init smu_init (void) INIT_LIST_HEAD(&smu->cmd_list); INIT_LIST_HEAD(&smu->cmd_i2c_list); smu->of_node = np; - smu->db_irq = NO_IRQ; - smu->msg_irq = NO_IRQ; + smu->db_irq = 0; + smu->msg_irq = 0; /* smu_cmdbuf_abs is in the low 2G of RAM, can be converted to a * 32 bits value safely @@ -587,13 +587,13 @@ static int smu_late_init(void) if (smu->db_node) { smu->db_irq = irq_of_parse_and_map(smu->db_node, 0); - if (smu->db_irq == NO_IRQ) + if (!smu->db_irq) printk(KERN_ERR "smu: failed to map irq for node %s\n", smu->db_node->full_name); } if (smu->msg_node) { smu->msg_irq = irq_of_parse_and_map(smu->msg_node, 0); - if (smu->msg_irq == NO_IRQ) + if (!smu->msg_irq) printk(KERN_ERR "smu: failed to map irq for node %s\n", smu->msg_node->full_name); } @@ -602,23 +602,23 @@ static int smu_late_init(void) * Try to request the interrupts */ - if (smu->db_irq != NO_IRQ) { + if (smu->db_irq) { if (request_irq(smu->db_irq, smu_db_intr, IRQF_SHARED, "SMU doorbell", smu) < 0) { printk(KERN_WARNING "SMU: can't " "request interrupt %d\n", smu->db_irq); - smu->db_irq = NO_IRQ; + smu->db_irq = 0; } } - if (smu->msg_irq != NO_IRQ) { + if (smu->msg_irq) { if (request_irq(smu->msg_irq, smu_msg_intr, IRQF_SHARED, "SMU message", smu) < 0) { printk(KERN_WARNING "SMU: can't " "request interrupt %d\n", smu->msg_irq); - smu->msg_irq = NO_IRQ; + smu->msg_irq = 0; } } diff --git a/drivers/macintosh/via-cuda.c b/drivers/macintosh/via-cuda.c index bad18130f125..2088e23a8002 100644 --- a/drivers/macintosh/via-cuda.c +++ b/drivers/macintosh/via-cuda.c @@ -209,7 +209,7 @@ static int __init via_cuda_start(void) cuda_irq = IRQ_MAC_ADB; #else cuda_irq = irq_of_parse_and_map(vias, 0); - if (cuda_irq == NO_IRQ) { + if (!cuda_irq) { printk(KERN_ERR "via-cuda: can't map interrupts for %s\n", vias->full_name); return -ENODEV; diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c index f8b6d1403c16..91081dcdc272 100644 --- a/drivers/macintosh/via-pmu.c +++ b/drivers/macintosh/via-pmu.c @@ -145,7 +145,7 @@ static int pmu_fully_inited; static int pmu_has_adb; static struct device_node *gpio_node; static unsigned char __iomem *gpio_reg; -static int gpio_irq = NO_IRQ; +static int gpio_irq = 0; static int gpio_irq_enabled = -1; static volatile int pmu_suspended; static spinlock_t pmu_lock; @@ -402,7 +402,7 @@ static int __init via_pmu_start(void) batt_req.complete = 1; irq = irq_of_parse_and_map(vias, 0); - if (irq == NO_IRQ) { + if (!irq) { printk(KERN_ERR "via-pmu: can't map interrupt\n"); return -ENODEV; } @@ -424,7 +424,7 @@ static int __init via_pmu_start(void) if (gpio_node) gpio_irq = irq_of_parse_and_map(gpio_node, 0); - if (gpio_irq != NO_IRQ) { + if (gpio_irq) { if (request_irq(gpio_irq, gpio1_interrupt, IRQF_NO_SUSPEND, "GPIO1 ADB", (void *)0)) diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h index 344a0ff8f8c7..01d372aba131 100644 --- a/drivers/misc/cxl/cxl.h +++ b/drivers/misc/cxl/cxl.h @@ -162,7 +162,10 @@ static const cxl_p2n_reg_t CXL_PSL_WED_An = {0x0A0}; #define CXL_PSL_SPAP_V 0x0000000000000001ULL /****** CXL_PSL_Control ****************************************************/ -#define CXL_PSL_Control_tb 0x0000000000000001ULL +#define CXL_PSL_Control_tb (0x1ull << (63-63)) +#define CXL_PSL_Control_Fr (0x1ull << (63-31)) +#define CXL_PSL_Control_Fs_MASK (0x3ull << (63-29)) +#define CXL_PSL_Control_Fs_Complete (0x3ull << (63-29)) /****** CXL_PSL_DLCNTL *****************************************************/ #define CXL_PSL_DLCNTL_D (0x1ull << (63-28)) @@ -854,6 +857,7 @@ int cxl_register_one_irq(struct cxl *adapter, irq_handler_t handler, int cxl_check_error(struct cxl_afu *afu); int cxl_afu_slbia(struct cxl_afu *afu); int cxl_tlb_slb_invalidate(struct cxl *adapter); +int cxl_data_cache_flush(struct cxl *adapter); int cxl_afu_disable(struct cxl_afu *afu); int cxl_psl_purge(struct cxl_afu *afu); diff --git a/drivers/misc/cxl/native.c b/drivers/misc/cxl/native.c index e606fdc4bc9c..a217a74ccc98 100644 --- a/drivers/misc/cxl/native.c +++ b/drivers/misc/cxl/native.c @@ -290,6 +290,37 @@ int cxl_tlb_slb_invalidate(struct cxl *adapter) return 0; } +int cxl_data_cache_flush(struct cxl *adapter) +{ + u64 reg; + unsigned long timeout = jiffies + (HZ * CXL_TIMEOUT); + + pr_devel("Flushing data cache\n"); + + reg = cxl_p1_read(adapter, CXL_PSL_Control); + reg |= CXL_PSL_Control_Fr; + cxl_p1_write(adapter, CXL_PSL_Control, reg); + + reg = cxl_p1_read(adapter, CXL_PSL_Control); + while ((reg & CXL_PSL_Control_Fs_MASK) != CXL_PSL_Control_Fs_Complete) { + if (time_after_eq(jiffies, timeout)) { + dev_warn(&adapter->dev, "WARNING: cache flush timed out!\n"); + return -EBUSY; + } + + if (!cxl_ops->link_ok(adapter, NULL)) { + dev_warn(&adapter->dev, "WARNING: link down when flushing cache\n"); + return -EIO; + } + cpu_relax(); + reg = cxl_p1_read(adapter, CXL_PSL_Control); + } + + reg &= ~CXL_PSL_Control_Fr; + cxl_p1_write(adapter, CXL_PSL_Control, reg); + return 0; +} + static int cxl_write_sstp(struct cxl_afu *afu, u64 sstp0, u64 sstp1) { int rc; diff --git a/drivers/misc/cxl/of.c b/drivers/misc/cxl/of.c index edc458395f68..ec175ea5dfba 100644 --- a/drivers/misc/cxl/of.c +++ b/drivers/misc/cxl/of.c @@ -460,7 +460,7 @@ int cxl_of_probe(struct platform_device *pdev) struct device_node *afu_np = NULL; struct cxl *adapter = NULL; int ret; - int slice, slice_ok; + int slice = 0, slice_ok = 0; pr_devel("in %s\n", __func__); @@ -476,13 +476,13 @@ int cxl_of_probe(struct platform_device *pdev) } /* init afu */ - slice_ok = 0; - for (afu_np = NULL, slice = 0; (afu_np = of_get_next_child(np, afu_np)); slice++) { + for_each_child_of_node(np, afu_np) { if ((ret = cxl_guest_init_afu(adapter, slice, afu_np))) dev_err(&pdev->dev, "AFU %i failed to initialise: %i\n", slice, ret); else slice_ok++; + slice++; } if (slice_ok == 0) { @@ -490,8 +490,6 @@ int cxl_of_probe(struct platform_device *pdev) adapter->slices = 0; } - if (afu_np) - of_node_put(afu_np); return 0; } diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c index 6f0c4ac4b649..7afad8477ad5 100644 --- a/drivers/misc/cxl/pci.c +++ b/drivers/misc/cxl/pci.c @@ -1239,6 +1239,9 @@ int cxl_pci_reset(struct cxl *adapter) dev_info(&dev->dev, "CXL reset\n"); + /* the adapter is about to be reset, so ignore errors */ + cxl_data_cache_flush(adapter); + /* pcie_warm_reset requests a fundamental pci reset which includes a * PERST assert/deassert. PERST triggers a loading of the image * if "user" or "factory" is selected in sysfs */ @@ -1530,11 +1533,11 @@ static void set_sl_ops(struct cxl *adapter, struct pci_dev *dev) { if (dev->vendor == PCI_VENDOR_ID_MELLANOX && dev->device == 0x1013) { /* Mellanox CX-4 */ - dev_info(&adapter->dev, "Device uses an XSL\n"); + dev_info(&dev->dev, "Device uses an XSL\n"); adapter->native->sl_ops = &xsl_ops; adapter->min_pe = 1; /* Workaround for CX-4 hardware bug */ } else { - dev_info(&adapter->dev, "Device uses a PSL\n"); + dev_info(&dev->dev, "Device uses a PSL\n"); adapter->native->sl_ops = &psl_ops; } } diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c index e6245b03f0a1..56efaf72d08e 100644 --- a/drivers/pci/hotplug/pnv_php.c +++ b/drivers/pci/hotplug/pnv_php.c @@ -22,6 +22,12 @@ #define DRIVER_AUTHOR "Gavin Shan, IBM Corporation" #define DRIVER_DESC "PowerPC PowerNV PCI Hotplug Driver" +struct pnv_php_event { + bool added; + struct pnv_php_slot *php_slot; + struct work_struct work; +}; + static LIST_HEAD(pnv_php_slot_list); static DEFINE_SPINLOCK(pnv_php_lock); @@ -29,12 +35,40 @@ static void pnv_php_register(struct device_node *dn); static void pnv_php_unregister_one(struct device_node *dn); static void pnv_php_unregister(struct device_node *dn); +static void pnv_php_disable_irq(struct pnv_php_slot *php_slot) +{ + struct pci_dev *pdev = php_slot->pdev; + u16 ctrl; + + if (php_slot->irq > 0) { + pcie_capability_read_word(pdev, PCI_EXP_SLTCTL, &ctrl); + ctrl &= ~(PCI_EXP_SLTCTL_HPIE | + PCI_EXP_SLTCTL_PDCE | + PCI_EXP_SLTCTL_DLLSCE); + pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, ctrl); + + free_irq(php_slot->irq, php_slot); + php_slot->irq = 0; + } + + if (php_slot->wq) { + destroy_workqueue(php_slot->wq); + php_slot->wq = NULL; + } + + if (pdev->msix_enabled) + pci_disable_msix(pdev); + else if (pdev->msi_enabled) + pci_disable_msi(pdev); +} + static void pnv_php_free_slot(struct kref *kref) { struct pnv_php_slot *php_slot = container_of(kref, struct pnv_php_slot, kref); WARN_ON(!list_empty(&php_slot->children)); + pnv_php_disable_irq(php_slot); kfree(php_slot->name); kfree(php_slot); } @@ -122,7 +156,7 @@ static void pnv_php_detach_device_nodes(struct device_node *parent) of_node_put(dn); refcount = atomic_read(&dn->kobj.kref.refcount); - if (unlikely(refcount != 1)) + if (refcount != 1) pr_warn("Invalid refcount %d on <%s>\n", refcount, of_node_full_name(dn)); @@ -184,11 +218,11 @@ static int pnv_php_populate_changeset(struct of_changeset *ocs, for_each_child_of_node(dn, child) { ret = of_changeset_attach_node(ocs, child); - if (unlikely(ret)) + if (ret) break; ret = pnv_php_populate_changeset(ocs, child); - if (unlikely(ret)) + if (ret) break; } @@ -201,7 +235,7 @@ static void *pnv_php_add_one_pdn(struct device_node *dn, void *data) struct pci_dn *pdn; pdn = pci_add_device_node_info(hose, dn); - if (unlikely(!pdn)) + if (!pdn) return ERR_PTR(-ENOMEM); return NULL; @@ -224,21 +258,21 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) * fits the real size. */ fdt1 = kzalloc(0x10000, GFP_KERNEL); - if (unlikely(!fdt1)) { + if (!fdt1) { ret = -ENOMEM; dev_warn(&php_slot->pdev->dev, "Cannot alloc FDT blob\n"); goto out; } ret = pnv_pci_get_device_tree(php_slot->dn->phandle, fdt1, 0x10000); - if (unlikely(ret)) { + if (ret) { dev_warn(&php_slot->pdev->dev, "Error %d getting FDT blob\n", ret); goto free_fdt1; } fdt = kzalloc(fdt_totalsize(fdt1), GFP_KERNEL); - if (unlikely(!fdt)) { + if (!fdt) { ret = -ENOMEM; dev_warn(&php_slot->pdev->dev, "Cannot %d bytes memory\n", fdt_totalsize(fdt1)); @@ -248,7 +282,7 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) /* Unflatten device tree blob */ memcpy(fdt, fdt1, fdt_totalsize(fdt1)); dt = of_fdt_unflatten_tree(fdt, php_slot->dn, NULL); - if (unlikely(!dt)) { + if (!dt) { ret = -EINVAL; dev_warn(&php_slot->pdev->dev, "Cannot unflatten FDT\n"); goto free_fdt; @@ -258,7 +292,7 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) of_changeset_init(&php_slot->ocs); pnv_php_reverse_nodes(php_slot->dn); ret = pnv_php_populate_changeset(&php_slot->ocs, php_slot->dn); - if (unlikely(ret)) { + if (ret) { pnv_php_reverse_nodes(php_slot->dn); dev_warn(&php_slot->pdev->dev, "Error %d populating changeset\n", ret); @@ -267,7 +301,7 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) php_slot->dn->child = NULL; ret = of_changeset_apply(&php_slot->ocs); - if (unlikely(ret)) { + if (ret) { dev_warn(&php_slot->pdev->dev, "Error %d applying changeset\n", ret); goto destroy_changeset; @@ -301,7 +335,7 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot, int ret; ret = pnv_pci_set_power_state(php_slot->id, state, &msg); - if (likely(ret > 0)) { + if (ret > 0) { if (be64_to_cpu(msg.params[1]) != php_slot->dn->phandle || be64_to_cpu(msg.params[2]) != state || be64_to_cpu(msg.params[3]) != OPAL_SUCCESS) { @@ -311,7 +345,7 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot, be64_to_cpu(msg.params[3])); return -ENOMSG; } - } else if (unlikely(ret < 0)) { + } else if (ret < 0) { dev_warn(&php_slot->pdev->dev, "Error %d powering %s\n", ret, (state == OPAL_PCI_SLOT_POWER_ON) ? "on" : "off"); return ret; @@ -338,7 +372,7 @@ static int pnv_php_get_power_state(struct hotplug_slot *slot, u8 *state) * be on. */ ret = pnv_pci_get_power_state(php_slot->id, &power_state); - if (unlikely(ret)) { + if (ret) { dev_warn(&php_slot->pdev->dev, "Error %d getting power status\n", ret); } else { @@ -360,7 +394,7 @@ static int pnv_php_get_adapter_state(struct hotplug_slot *slot, u8 *state) * get that, it will fail back to be empty. */ ret = pnv_pci_get_presence_state(php_slot->id, &presence); - if (likely(ret >= 0)) { + if (ret >= 0) { *state = presence; slot->info->adapter_status = presence; ret = 0; @@ -393,7 +427,7 @@ static int pnv_php_enable(struct pnv_php_slot *php_slot, bool rescan) /* Retrieve slot presence status */ ret = pnv_php_get_adapter_state(slot, &presence); - if (unlikely(ret)) + if (ret) return ret; /* Proceed if there have nothing behind the slot */ @@ -414,7 +448,7 @@ static int pnv_php_enable(struct pnv_php_slot *php_slot, bool rescan) php_slot->power_state_check = true; ret = pnv_php_get_power_state(slot, &power_status); - if (unlikely(ret)) + if (ret) return ret; if (power_status != OPAL_PCI_SLOT_POWER_ON) @@ -423,7 +457,7 @@ static int pnv_php_enable(struct pnv_php_slot *php_slot, bool rescan) /* Check the power status. Scan the slot if it is already on */ ret = pnv_php_get_power_state(slot, &power_status); - if (unlikely(ret)) + if (ret) return ret; if (power_status == OPAL_PCI_SLOT_POWER_ON) @@ -431,7 +465,7 @@ static int pnv_php_enable(struct pnv_php_slot *php_slot, bool rescan) /* Power is off, turn it on and then scan the slot */ ret = pnv_php_set_slot_power_state(slot, OPAL_PCI_SLOT_POWER_ON); - if (unlikely(ret)) + if (ret) return ret; scan: @@ -513,29 +547,30 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn) struct pci_bus *bus; const char *label; uint64_t id; + int ret; - label = of_get_property(dn, "ibm,slot-label", NULL); - if (unlikely(!label)) + ret = of_property_read_string(dn, "ibm,slot-label", &label); + if (ret) return NULL; if (pnv_pci_get_slot_id(dn, &id)) return NULL; bus = pci_find_bus_by_node(dn); - if (unlikely(!bus)) + if (!bus) return NULL; php_slot = kzalloc(sizeof(*php_slot), GFP_KERNEL); - if (unlikely(!php_slot)) + if (!php_slot) return NULL; php_slot->name = kstrdup(label, GFP_KERNEL); - if (unlikely(!php_slot->name)) { + if (!php_slot->name) { kfree(php_slot); return NULL; } - if (likely(dn->child && PCI_DN(dn->child))) + if (dn->child && PCI_DN(dn->child)) php_slot->slot_no = PCI_SLOT(PCI_DN(dn->child)->devfn); else php_slot->slot_no = -1; /* Placeholder slot */ @@ -567,7 +602,7 @@ static int pnv_php_register_slot(struct pnv_php_slot *php_slot) /* Check if the slot is registered or not */ parent = pnv_php_find_slot(php_slot->dn); - if (unlikely(parent)) { + if (parent) { pnv_php_put_slot(parent); return -EEXIST; } @@ -575,7 +610,7 @@ static int pnv_php_register_slot(struct pnv_php_slot *php_slot) /* Register PCI slot */ ret = pci_hp_register(&php_slot->slot, php_slot->bus, php_slot->slot_no, php_slot->name); - if (unlikely(ret)) { + if (ret) { dev_warn(&php_slot->pdev->dev, "Error %d registering slot\n", ret); return ret; @@ -609,33 +644,213 @@ static int pnv_php_register_slot(struct pnv_php_slot *php_slot) return 0; } +static int pnv_php_enable_msix(struct pnv_php_slot *php_slot) +{ + struct pci_dev *pdev = php_slot->pdev; + struct msix_entry entry; + int nr_entries, ret; + u16 pcie_flag; + + /* Get total number of MSIx entries */ + nr_entries = pci_msix_vec_count(pdev); + if (nr_entries < 0) + return nr_entries; + + /* Check hotplug MSIx entry is in range */ + pcie_capability_read_word(pdev, PCI_EXP_FLAGS, &pcie_flag); + entry.entry = (pcie_flag & PCI_EXP_FLAGS_IRQ) >> 9; + if (entry.entry >= nr_entries) + return -ERANGE; + + /* Enable MSIx */ + ret = pci_enable_msix_exact(pdev, &entry, 1); + if (ret) { + dev_warn(&pdev->dev, "Error %d enabling MSIx\n", ret); + return ret; + } + + return entry.vector; +} + +static void pnv_php_event_handler(struct work_struct *work) +{ + struct pnv_php_event *event = + container_of(work, struct pnv_php_event, work); + struct pnv_php_slot *php_slot = event->php_slot; + + if (event->added) + pnv_php_enable_slot(&php_slot->slot); + else + pnv_php_disable_slot(&php_slot->slot); + + kfree(event); +} + +static irqreturn_t pnv_php_interrupt(int irq, void *data) +{ + struct pnv_php_slot *php_slot = data; + struct pci_dev *pchild, *pdev = php_slot->pdev; + struct eeh_dev *edev; + struct eeh_pe *pe; + struct pnv_php_event *event; + u16 sts, lsts; + u8 presence; + bool added; + unsigned long flags; + int ret; + + pcie_capability_read_word(pdev, PCI_EXP_SLTSTA, &sts); + sts &= (PCI_EXP_SLTSTA_PDC | PCI_EXP_SLTSTA_DLLSC); + pcie_capability_write_word(pdev, PCI_EXP_SLTSTA, sts); + if (sts & PCI_EXP_SLTSTA_DLLSC) { + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lsts); + added = !!(lsts & PCI_EXP_LNKSTA_DLLLA); + } else if (sts & PCI_EXP_SLTSTA_PDC) { + ret = pnv_pci_get_presence_state(php_slot->id, &presence); + if (!ret) + return IRQ_HANDLED; + added = !!(presence == OPAL_PCI_SLOT_PRESENT); + } else { + return IRQ_NONE; + } + + /* Freeze the removed PE to avoid unexpected error reporting */ + if (!added) { + pchild = list_first_entry_or_null(&php_slot->bus->devices, + struct pci_dev, bus_list); + edev = pchild ? pci_dev_to_eeh_dev(pchild) : NULL; + pe = edev ? edev->pe : NULL; + if (pe) { + eeh_serialize_lock(&flags); + eeh_pe_state_mark(pe, EEH_PE_ISOLATED); + eeh_serialize_unlock(flags); + eeh_pe_set_option(pe, EEH_OPT_FREEZE_PE); + } + } + + /* + * The PE is left in frozen state if the event is missed. It's + * fine as the PCI devices (PE) aren't functional any more. + */ + event = kzalloc(sizeof(*event), GFP_ATOMIC); + if (!event) { + dev_warn(&pdev->dev, "PCI slot [%s] missed hotplug event 0x%04x\n", + php_slot->name, sts); + return IRQ_HANDLED; + } + + dev_info(&pdev->dev, "PCI slot [%s] %s (IRQ: %d)\n", + php_slot->name, added ? "added" : "removed", irq); + INIT_WORK(&event->work, pnv_php_event_handler); + event->added = added; + event->php_slot = php_slot; + queue_work(php_slot->wq, &event->work); + + return IRQ_HANDLED; +} + +static void pnv_php_init_irq(struct pnv_php_slot *php_slot, int irq) +{ + struct pci_dev *pdev = php_slot->pdev; + u16 sts, ctrl; + int ret; + + /* Allocate workqueue */ + php_slot->wq = alloc_workqueue("pciehp-%s", 0, 0, php_slot->name); + if (!php_slot->wq) { + dev_warn(&pdev->dev, "Cannot alloc workqueue\n"); + pnv_php_disable_irq(php_slot); + return; + } + + /* Clear pending interrupts */ + pcie_capability_read_word(pdev, PCI_EXP_SLTSTA, &sts); + sts |= (PCI_EXP_SLTSTA_PDC | PCI_EXP_SLTSTA_DLLSC); + pcie_capability_write_word(pdev, PCI_EXP_SLTSTA, sts); + + /* Request the interrupt */ + ret = request_irq(irq, pnv_php_interrupt, IRQF_SHARED, + php_slot->name, php_slot); + if (ret) { + pnv_php_disable_irq(php_slot); + dev_warn(&pdev->dev, "Error %d enabling IRQ %d\n", ret, irq); + return; + } + + /* Enable the interrupts */ + pcie_capability_read_word(pdev, PCI_EXP_SLTCTL, &ctrl); + ctrl |= (PCI_EXP_SLTCTL_HPIE | + PCI_EXP_SLTCTL_PDCE | + PCI_EXP_SLTCTL_DLLSCE); + pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, ctrl); + + /* The interrupt is initialized successfully when @irq is valid */ + php_slot->irq = irq; +} + +static void pnv_php_enable_irq(struct pnv_php_slot *php_slot) +{ + struct pci_dev *pdev = php_slot->pdev; + int irq, ret; + + ret = pci_enable_device(pdev); + if (ret) { + dev_warn(&pdev->dev, "Error %d enabling device\n", ret); + return; + } + + pci_set_master(pdev); + + /* Enable MSIx interrupt */ + irq = pnv_php_enable_msix(php_slot); + if (irq > 0) { + pnv_php_init_irq(php_slot, irq); + return; + } + + /* + * Use MSI if MSIx doesn't work. Fail back to legacy INTx + * if MSI doesn't work either + */ + ret = pci_enable_msi(pdev); + if (!ret || pdev->irq) { + irq = pdev->irq; + pnv_php_init_irq(php_slot, irq); + } +} + static int pnv_php_register_one(struct device_node *dn) { struct pnv_php_slot *php_slot; - const __be32 *prop32; + u32 prop32; int ret; /* Check if it's hotpluggable slot */ - prop32 = of_get_property(dn, "ibm,slot-pluggable", NULL); - if (!prop32 || !of_read_number(prop32, 1)) + ret = of_property_read_u32(dn, "ibm,slot-pluggable", &prop32); + if (ret || !prop32) return -ENXIO; - prop32 = of_get_property(dn, "ibm,reset-by-firmware", NULL); - if (!prop32 || !of_read_number(prop32, 1)) + ret = of_property_read_u32(dn, "ibm,reset-by-firmware", &prop32); + if (ret || !prop32) return -ENXIO; php_slot = pnv_php_alloc_slot(dn); - if (unlikely(!php_slot)) + if (!php_slot) return -ENODEV; ret = pnv_php_register_slot(php_slot); - if (unlikely(ret)) + if (ret) goto free_slot; ret = pnv_php_enable(php_slot, false); - if (unlikely(ret)) + if (ret) goto unregister_slot; + /* Enable interrupt if the slot supports surprise hotplug */ + ret = of_property_read_u32(dn, "ibm,slot-surprise-pluggable", &prop32); + if (!ret && prop32) + pnv_php_enable_irq(php_slot); + return 0; unregister_slot: diff --git a/drivers/ps3/ps3-vuart.c b/drivers/ps3/ps3-vuart.c index 632701a1d993..b7f300b79ffd 100644 --- a/drivers/ps3/ps3-vuart.c +++ b/drivers/ps3/ps3-vuart.c @@ -958,7 +958,7 @@ static int ps3_vuart_bus_interrupt_get(void) fail_request_irq: ps3_vuart_irq_destroy(vuart_bus_priv.virq); - vuart_bus_priv.virq = NO_IRQ; + vuart_bus_priv.virq = 0; fail_alloc_irq: kfree(vuart_bus_priv.bmp); vuart_bus_priv.bmp = NULL; @@ -982,7 +982,7 @@ static int ps3_vuart_bus_interrupt_put(void) free_irq(vuart_bus_priv.virq, &vuart_bus_priv); ps3_vuart_irq_destroy(vuart_bus_priv.virq); - vuart_bus_priv.virq = NO_IRQ; + vuart_bus_priv.virq = 0; kfree(vuart_bus_priv.bmp); vuart_bus_priv.bmp = NULL; diff --git a/sound/aoa/core/gpio-feature.c b/sound/aoa/core/gpio-feature.c index f34153962d07..71960089e207 100644 --- a/sound/aoa/core/gpio-feature.c +++ b/sound/aoa/core/gpio-feature.c @@ -118,7 +118,7 @@ static void get_irq(struct device_node * np, int *irqptr) if (np) *irqptr = irq_of_parse_and_map(np, 0); else - *irqptr = NO_IRQ; + *irqptr = 0; } /* 0x4 is outenable, 0x1 is out, thus 4 or 5 */ @@ -336,7 +336,7 @@ static int ftr_set_notify(struct gpio_runtime *rt, return -EINVAL; } - if (irq == NO_IRQ) + if (!irq) return -ENODEV; mutex_lock(¬if->mutex); diff --git a/sound/ppc/tumbler.c b/sound/ppc/tumbler.c index c8fafba218a5..58ee8089bbf9 100644 --- a/sound/ppc/tumbler.c +++ b/sound/ppc/tumbler.c @@ -1303,19 +1303,19 @@ static int tumbler_init(struct snd_pmac *chip) &mix->line_mute, 1); irq = tumbler_find_device("headphone-detect", NULL, &mix->hp_detect, 0); - if (irq <= NO_IRQ) + if (irq <= 0) irq = tumbler_find_device("headphone-detect", NULL, &mix->hp_detect, 1); - if (irq <= NO_IRQ) + if (irq <= 0) irq = tumbler_find_device("keywest-gpio15", NULL, &mix->hp_detect, 1); mix->headphone_irq = irq; irq = tumbler_find_device("line-output-detect", NULL, &mix->line_detect, 0); - if (irq <= NO_IRQ) + if (irq <= 0) irq = tumbler_find_device("line-output-detect", NULL, &mix->line_detect, 1); - if (IS_G4DA && irq <= NO_IRQ) + if (IS_G4DA && irq <= 0) irq = tumbler_find_device("keywest-gpio16", NULL, &mix->line_detect, 1); mix->lineout_irq = irq; diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile index 1cc6d64c39b7..db54a33f850f 100644 --- a/tools/testing/selftests/powerpc/Makefile +++ b/tools/testing/selftests/powerpc/Makefile @@ -19,6 +19,7 @@ SUB_DIRS = alignment \ dscr \ mm \ pmu \ + signal \ primitives \ stringloops \ switch_endian \ diff --git a/tools/testing/selftests/powerpc/fpu_asm.h b/tools/testing/selftests/powerpc/fpu_asm.h new file mode 100644 index 000000000000..6a387d255e27 --- /dev/null +++ b/tools/testing/selftests/powerpc/fpu_asm.h @@ -0,0 +1,80 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#ifndef _SELFTESTS_POWERPC_FPU_ASM_H +#define _SELFTESTS_POWERPC_FPU_ASM_H +#include "basic_asm.h" + +#define PUSH_FPU(stack_size) \ + stfd f31,(stack_size + STACK_FRAME_MIN_SIZE)(%r1); \ + stfd f30,(stack_size + STACK_FRAME_MIN_SIZE - 8)(%r1); \ + stfd f29,(stack_size + STACK_FRAME_MIN_SIZE - 16)(%r1); \ + stfd f28,(stack_size + STACK_FRAME_MIN_SIZE - 24)(%r1); \ + stfd f27,(stack_size + STACK_FRAME_MIN_SIZE - 32)(%r1); \ + stfd f26,(stack_size + STACK_FRAME_MIN_SIZE - 40)(%r1); \ + stfd f25,(stack_size + STACK_FRAME_MIN_SIZE - 48)(%r1); \ + stfd f24,(stack_size + STACK_FRAME_MIN_SIZE - 56)(%r1); \ + stfd f23,(stack_size + STACK_FRAME_MIN_SIZE - 64)(%r1); \ + stfd f22,(stack_size + STACK_FRAME_MIN_SIZE - 72)(%r1); \ + stfd f21,(stack_size + STACK_FRAME_MIN_SIZE - 80)(%r1); \ + stfd f20,(stack_size + STACK_FRAME_MIN_SIZE - 88)(%r1); \ + stfd f19,(stack_size + STACK_FRAME_MIN_SIZE - 96)(%r1); \ + stfd f18,(stack_size + STACK_FRAME_MIN_SIZE - 104)(%r1); \ + stfd f17,(stack_size + STACK_FRAME_MIN_SIZE - 112)(%r1); \ + stfd f16,(stack_size + STACK_FRAME_MIN_SIZE - 120)(%r1); \ + stfd f15,(stack_size + STACK_FRAME_MIN_SIZE - 128)(%r1); \ + stfd f14,(stack_size + STACK_FRAME_MIN_SIZE - 136)(%r1); + +#define POP_FPU(stack_size) \ + lfd f31,(stack_size + STACK_FRAME_MIN_SIZE)(%r1); \ + lfd f30,(stack_size + STACK_FRAME_MIN_SIZE - 8)(%r1); \ + lfd f29,(stack_size + STACK_FRAME_MIN_SIZE - 16)(%r1); \ + lfd f28,(stack_size + STACK_FRAME_MIN_SIZE - 24)(%r1); \ + lfd f27,(stack_size + STACK_FRAME_MIN_SIZE - 32)(%r1); \ + lfd f26,(stack_size + STACK_FRAME_MIN_SIZE - 40)(%r1); \ + lfd f25,(stack_size + STACK_FRAME_MIN_SIZE - 48)(%r1); \ + lfd f24,(stack_size + STACK_FRAME_MIN_SIZE - 56)(%r1); \ + lfd f23,(stack_size + STACK_FRAME_MIN_SIZE - 64)(%r1); \ + lfd f22,(stack_size + STACK_FRAME_MIN_SIZE - 72)(%r1); \ + lfd f21,(stack_size + STACK_FRAME_MIN_SIZE - 80)(%r1); \ + lfd f20,(stack_size + STACK_FRAME_MIN_SIZE - 88)(%r1); \ + lfd f19,(stack_size + STACK_FRAME_MIN_SIZE - 96)(%r1); \ + lfd f18,(stack_size + STACK_FRAME_MIN_SIZE - 104)(%r1); \ + lfd f17,(stack_size + STACK_FRAME_MIN_SIZE - 112)(%r1); \ + lfd f16,(stack_size + STACK_FRAME_MIN_SIZE - 120)(%r1); \ + lfd f15,(stack_size + STACK_FRAME_MIN_SIZE - 128)(%r1); \ + lfd f14,(stack_size + STACK_FRAME_MIN_SIZE - 136)(%r1); + +/* + * Careful calling this, it will 'clobber' fpu (by design) + * Don't call this from C + */ +FUNC_START(load_fpu) + lfd f14,0(r3) + lfd f15,8(r3) + lfd f16,16(r3) + lfd f17,24(r3) + lfd f18,32(r3) + lfd f19,40(r3) + lfd f20,48(r3) + lfd f21,56(r3) + lfd f22,64(r3) + lfd f23,72(r3) + lfd f24,80(r3) + lfd f25,88(r3) + lfd f26,96(r3) + lfd f27,104(r3) + lfd f28,112(r3) + lfd f29,120(r3) + lfd f30,128(r3) + lfd f31,136(r3) + blr +FUNC_END(load_fpu) + +#endif /* _SELFTESTS_POWERPC_FPU_ASM_H */ diff --git a/tools/testing/selftests/powerpc/gpr_asm.h b/tools/testing/selftests/powerpc/gpr_asm.h new file mode 100644 index 000000000000..f6f38852d3a0 --- /dev/null +++ b/tools/testing/selftests/powerpc/gpr_asm.h @@ -0,0 +1,96 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#ifndef _SELFTESTS_POWERPC_GPR_ASM_H +#define _SELFTESTS_POWERPC_GPR_ASM_H + +#include "basic_asm.h" + +#define __PUSH_NVREGS(top_pos); \ + std r31,(top_pos)(%r1); \ + std r30,(top_pos - 8)(%r1); \ + std r29,(top_pos - 16)(%r1); \ + std r28,(top_pos - 24)(%r1); \ + std r27,(top_pos - 32)(%r1); \ + std r26,(top_pos - 40)(%r1); \ + std r25,(top_pos - 48)(%r1); \ + std r24,(top_pos - 56)(%r1); \ + std r23,(top_pos - 64)(%r1); \ + std r22,(top_pos - 72)(%r1); \ + std r21,(top_pos - 80)(%r1); \ + std r20,(top_pos - 88)(%r1); \ + std r19,(top_pos - 96)(%r1); \ + std r18,(top_pos - 104)(%r1); \ + std r17,(top_pos - 112)(%r1); \ + std r16,(top_pos - 120)(%r1); \ + std r15,(top_pos - 128)(%r1); \ + std r14,(top_pos - 136)(%r1) + +#define __POP_NVREGS(top_pos); \ + ld r31,(top_pos)(%r1); \ + ld r30,(top_pos - 8)(%r1); \ + ld r29,(top_pos - 16)(%r1); \ + ld r28,(top_pos - 24)(%r1); \ + ld r27,(top_pos - 32)(%r1); \ + ld r26,(top_pos - 40)(%r1); \ + ld r25,(top_pos - 48)(%r1); \ + ld r24,(top_pos - 56)(%r1); \ + ld r23,(top_pos - 64)(%r1); \ + ld r22,(top_pos - 72)(%r1); \ + ld r21,(top_pos - 80)(%r1); \ + ld r20,(top_pos - 88)(%r1); \ + ld r19,(top_pos - 96)(%r1); \ + ld r18,(top_pos - 104)(%r1); \ + ld r17,(top_pos - 112)(%r1); \ + ld r16,(top_pos - 120)(%r1); \ + ld r15,(top_pos - 128)(%r1); \ + ld r14,(top_pos - 136)(%r1) + +#define PUSH_NVREGS(stack_size) \ + __PUSH_NVREGS(stack_size + STACK_FRAME_MIN_SIZE) + +/* 18 NV FPU REGS */ +#define PUSH_NVREGS_BELOW_FPU(stack_size) \ + __PUSH_NVREGS(stack_size + STACK_FRAME_MIN_SIZE - (18 * 8)) + +#define POP_NVREGS(stack_size) \ + __POP_NVREGS(stack_size + STACK_FRAME_MIN_SIZE) + +/* 18 NV FPU REGS */ +#define POP_NVREGS_BELOW_FPU(stack_size) \ + __POP_NVREGS(stack_size + STACK_FRAME_MIN_SIZE - (18 * 8)) + +/* + * Careful calling this, it will 'clobber' NVGPRs (by design) + * Don't call this from C + */ +FUNC_START(load_gpr) + ld r14,0(r3) + ld r15,8(r3) + ld r16,16(r3) + ld r17,24(r3) + ld r18,32(r3) + ld r19,40(r3) + ld r20,48(r3) + ld r21,56(r3) + ld r22,64(r3) + ld r23,72(r3) + ld r24,80(r3) + ld r25,88(r3) + ld r26,96(r3) + ld r27,104(r3) + ld r28,112(r3) + ld r29,120(r3) + ld r30,128(r3) + ld r31,136(r3) + blr +FUNC_END(load_gpr) + + +#endif /* _SELFTESTS_POWERPC_GPR_ASM_H */ diff --git a/tools/testing/selftests/powerpc/harness.c b/tools/testing/selftests/powerpc/harness.c index 52f9be7f61f0..248a820048df 100644 --- a/tools/testing/selftests/powerpc/harness.c +++ b/tools/testing/selftests/powerpc/harness.c @@ -19,9 +19,9 @@ #include "subunit.h" #include "utils.h" -#define TIMEOUT 120 #define KILL_TIMEOUT 5 +static uint64_t timeout = 120; int run_test(int (test_function)(void), char *name) { @@ -44,7 +44,7 @@ int run_test(int (test_function)(void), char *name) setpgid(pid, pid); /* Wake us up in timeout seconds */ - alarm(TIMEOUT); + alarm(timeout); terminated = false; wait: @@ -94,6 +94,11 @@ static struct sigaction alarm_action = { .sa_handler = alarm_handler, }; +void test_harness_set_timeout(uint64_t time) +{ + timeout = time; +} + int test_harness(int (test_function)(void), char *name) { int rc; diff --git a/tools/testing/selftests/powerpc/math/Makefile b/tools/testing/selftests/powerpc/math/Makefile index 5b88875d5955..a505b66d408a 100644 --- a/tools/testing/selftests/powerpc/math/Makefile +++ b/tools/testing/selftests/powerpc/math/Makefile @@ -1,4 +1,4 @@ -TEST_PROGS := fpu_syscall fpu_preempt fpu_signal vmx_syscall vmx_preempt vmx_signal +TEST_PROGS := fpu_syscall fpu_preempt fpu_signal vmx_syscall vmx_preempt vmx_signal vsx_preempt all: $(TEST_PROGS) @@ -13,6 +13,9 @@ vmx_syscall: vmx_asm.S vmx_preempt: vmx_asm.S vmx_signal: vmx_asm.S +vsx_preempt: CFLAGS += -mvsx +vsx_preempt: vsx_asm.S + include ../../lib.mk clean: diff --git a/tools/testing/selftests/powerpc/math/fpu_asm.S b/tools/testing/selftests/powerpc/math/fpu_asm.S index f3711d80e709..241f067a510f 100644 --- a/tools/testing/selftests/powerpc/math/fpu_asm.S +++ b/tools/testing/selftests/powerpc/math/fpu_asm.S @@ -8,70 +8,7 @@ */ #include "../basic_asm.h" - -#define PUSH_FPU(pos) \ - stfd f14,pos(sp); \ - stfd f15,pos+8(sp); \ - stfd f16,pos+16(sp); \ - stfd f17,pos+24(sp); \ - stfd f18,pos+32(sp); \ - stfd f19,pos+40(sp); \ - stfd f20,pos+48(sp); \ - stfd f21,pos+56(sp); \ - stfd f22,pos+64(sp); \ - stfd f23,pos+72(sp); \ - stfd f24,pos+80(sp); \ - stfd f25,pos+88(sp); \ - stfd f26,pos+96(sp); \ - stfd f27,pos+104(sp); \ - stfd f28,pos+112(sp); \ - stfd f29,pos+120(sp); \ - stfd f30,pos+128(sp); \ - stfd f31,pos+136(sp); - -#define POP_FPU(pos) \ - lfd f14,pos(sp); \ - lfd f15,pos+8(sp); \ - lfd f16,pos+16(sp); \ - lfd f17,pos+24(sp); \ - lfd f18,pos+32(sp); \ - lfd f19,pos+40(sp); \ - lfd f20,pos+48(sp); \ - lfd f21,pos+56(sp); \ - lfd f22,pos+64(sp); \ - lfd f23,pos+72(sp); \ - lfd f24,pos+80(sp); \ - lfd f25,pos+88(sp); \ - lfd f26,pos+96(sp); \ - lfd f27,pos+104(sp); \ - lfd f28,pos+112(sp); \ - lfd f29,pos+120(sp); \ - lfd f30,pos+128(sp); \ - lfd f31,pos+136(sp); - -# Careful calling this, it will 'clobber' fpu (by design) -# Don't call this from C -FUNC_START(load_fpu) - lfd f14,0(r3) - lfd f15,8(r3) - lfd f16,16(r3) - lfd f17,24(r3) - lfd f18,32(r3) - lfd f19,40(r3) - lfd f20,48(r3) - lfd f21,56(r3) - lfd f22,64(r3) - lfd f23,72(r3) - lfd f24,80(r3) - lfd f25,88(r3) - lfd f26,96(r3) - lfd f27,104(r3) - lfd f28,112(r3) - lfd f29,120(r3) - lfd f30,128(r3) - lfd f31,136(r3) - blr -FUNC_END(load_fpu) +#include "../fpu_asm.h" FUNC_START(check_fpu) mr r4,r3 @@ -138,9 +75,9 @@ FUNC_START(test_fpu) # r4 holds pointer to the pid # f14-f31 are non volatiles PUSH_BASIC_STACK(256) + PUSH_FPU(256) std r3,STACK_FRAME_PARAM(0)(sp) # Address of darray std r4,STACK_FRAME_PARAM(1)(sp) # Address of pid - PUSH_FPU(STACK_FRAME_LOCAL(2,0)) bl load_fpu nop @@ -155,7 +92,7 @@ FUNC_START(test_fpu) bl check_fpu nop - POP_FPU(STACK_FRAME_LOCAL(2,0)) + POP_FPU(256) POP_BASIC_STACK(256) blr FUNC_END(test_fpu) @@ -166,10 +103,10 @@ FUNC_END(test_fpu) # registers while running is not zero. FUNC_START(preempt_fpu) PUSH_BASIC_STACK(256) + PUSH_FPU(256) std r3,STACK_FRAME_PARAM(0)(sp) # double *darray std r4,STACK_FRAME_PARAM(1)(sp) # int *threads_starting std r5,STACK_FRAME_PARAM(2)(sp) # int *running - PUSH_FPU(STACK_FRAME_LOCAL(3,0)) bl load_fpu nop @@ -192,7 +129,7 @@ FUNC_START(preempt_fpu) cmpwi r5,0 bne 2b -3: POP_FPU(STACK_FRAME_LOCAL(3,0)) +3: POP_FPU(256) POP_BASIC_STACK(256) blr FUNC_END(preempt_fpu) diff --git a/tools/testing/selftests/powerpc/math/vmx_asm.S b/tools/testing/selftests/powerpc/math/vmx_asm.S index 1b8c248b3ac1..fd74da488625 100644 --- a/tools/testing/selftests/powerpc/math/vmx_asm.S +++ b/tools/testing/selftests/powerpc/math/vmx_asm.S @@ -8,90 +8,7 @@ */ #include "../basic_asm.h" - -# POS MUST BE 16 ALIGNED! -#define PUSH_VMX(pos,reg) \ - li reg,pos; \ - stvx v20,reg,sp; \ - addi reg,reg,16; \ - stvx v21,reg,sp; \ - addi reg,reg,16; \ - stvx v22,reg,sp; \ - addi reg,reg,16; \ - stvx v23,reg,sp; \ - addi reg,reg,16; \ - stvx v24,reg,sp; \ - addi reg,reg,16; \ - stvx v25,reg,sp; \ - addi reg,reg,16; \ - stvx v26,reg,sp; \ - addi reg,reg,16; \ - stvx v27,reg,sp; \ - addi reg,reg,16; \ - stvx v28,reg,sp; \ - addi reg,reg,16; \ - stvx v29,reg,sp; \ - addi reg,reg,16; \ - stvx v30,reg,sp; \ - addi reg,reg,16; \ - stvx v31,reg,sp; - -# POS MUST BE 16 ALIGNED! -#define POP_VMX(pos,reg) \ - li reg,pos; \ - lvx v20,reg,sp; \ - addi reg,reg,16; \ - lvx v21,reg,sp; \ - addi reg,reg,16; \ - lvx v22,reg,sp; \ - addi reg,reg,16; \ - lvx v23,reg,sp; \ - addi reg,reg,16; \ - lvx v24,reg,sp; \ - addi reg,reg,16; \ - lvx v25,reg,sp; \ - addi reg,reg,16; \ - lvx v26,reg,sp; \ - addi reg,reg,16; \ - lvx v27,reg,sp; \ - addi reg,reg,16; \ - lvx v28,reg,sp; \ - addi reg,reg,16; \ - lvx v29,reg,sp; \ - addi reg,reg,16; \ - lvx v30,reg,sp; \ - addi reg,reg,16; \ - lvx v31,reg,sp; - -# Carefull this will 'clobber' vmx (by design) -# Don't call this from C -FUNC_START(load_vmx) - li r5,0 - lvx v20,r5,r3 - addi r5,r5,16 - lvx v21,r5,r3 - addi r5,r5,16 - lvx v22,r5,r3 - addi r5,r5,16 - lvx v23,r5,r3 - addi r5,r5,16 - lvx v24,r5,r3 - addi r5,r5,16 - lvx v25,r5,r3 - addi r5,r5,16 - lvx v26,r5,r3 - addi r5,r5,16 - lvx v27,r5,r3 - addi r5,r5,16 - lvx v28,r5,r3 - addi r5,r5,16 - lvx v29,r5,r3 - addi r5,r5,16 - lvx v30,r5,r3 - addi r5,r5,16 - lvx v31,r5,r3 - blr -FUNC_END(load_vmx) +#include "../vmx_asm.h" # Should be safe from C, only touches r4, r5 and v0,v1,v2 FUNC_START(check_vmx) diff --git a/tools/testing/selftests/powerpc/math/vsx_asm.S b/tools/testing/selftests/powerpc/math/vsx_asm.S new file mode 100644 index 000000000000..a110dd882d5e --- /dev/null +++ b/tools/testing/selftests/powerpc/math/vsx_asm.S @@ -0,0 +1,61 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "../basic_asm.h" +#include "../vsx_asm.h" + +#long check_vsx(vector int *r3); +#This function wraps storeing VSX regs to the end of an array and a +#call to a comparison function in C which boils down to a memcmp() +FUNC_START(check_vsx) + PUSH_BASIC_STACK(32) + std r3,STACK_FRAME_PARAM(0)(sp) + addi r3, r3, 16 * 12 #Second half of array + bl store_vsx + ld r3,STACK_FRAME_PARAM(0)(sp) + bl vsx_memcmp + POP_BASIC_STACK(32) + blr +FUNC_END(check_vsx) + +# int preempt_vmx(vector int *varray, int *threads_starting, +# int *running); +# On starting will (atomically) decrement threads_starting as a signal +# that the VMX have been loaded with varray. Will proceed to check the +# validity of the VMX registers while running is not zero. +FUNC_START(preempt_vsx) + PUSH_BASIC_STACK(512) + std r3,STACK_FRAME_PARAM(0)(sp) # vector int *varray + std r4,STACK_FRAME_PARAM(1)(sp) # int *threads_starting + std r5,STACK_FRAME_PARAM(2)(sp) # int *running + + bl load_vsx + nop + + sync + # Atomic DEC + ld r3,STACK_FRAME_PARAM(1)(sp) +1: lwarx r4,0,r3 + addi r4,r4,-1 + stwcx. r4,0,r3 + bne- 1b + +2: ld r3,STACK_FRAME_PARAM(0)(sp) + bl check_vsx + nop + cmpdi r3,0 + bne 3f + ld r4,STACK_FRAME_PARAM(2)(sp) + ld r5,0(r4) + cmpwi r5,0 + bne 2b + +3: POP_BASIC_STACK(512) + blr +FUNC_END(preempt_vsx) diff --git a/tools/testing/selftests/powerpc/math/vsx_preempt.c b/tools/testing/selftests/powerpc/math/vsx_preempt.c new file mode 100644 index 000000000000..6387f03a0a6a --- /dev/null +++ b/tools/testing/selftests/powerpc/math/vsx_preempt.c @@ -0,0 +1,147 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * This test attempts to see if the VSX registers change across preemption. + * There is no way to be sure preemption happened so this test just + * uses many threads and a long wait. As such, a successful test + * doesn't mean much but a failure is bad. + */ + +#include <stdio.h> +#include <string.h> +#include <unistd.h> +#include <sys/syscall.h> +#include <sys/time.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <stdlib.h> +#include <pthread.h> + +#include "utils.h" + +/* Time to wait for workers to get preempted (seconds) */ +#define PREEMPT_TIME 20 +/* + * Factor by which to multiply number of online CPUs for total number of + * worker threads + */ +#define THREAD_FACTOR 8 + +/* + * Ensure there is twice the number of non-volatile VMX regs! + * check_vmx() is going to use the other half as space to put the live + * registers before calling vsx_memcmp() + */ +__thread vector int varray[24] = { + {1, 2, 3, 4 }, {5, 6, 7, 8 }, {9, 10,11,12}, + {13,14,15,16}, {17,18,19,20}, {21,22,23,24}, + {25,26,27,28}, {29,30,31,32}, {33,34,35,36}, + {37,38,39,40}, {41,42,43,44}, {45,46,47,48} +}; + +int threads_starting; +int running; + +extern long preempt_vsx(vector int *varray, int *threads_starting, int *running); + +long vsx_memcmp(vector int *a) { + vector int zero = {0, 0, 0, 0}; + int i; + + FAIL_IF(a != varray); + + for(i = 0; i < 12; i++) { + if (memcmp(&a[i + 12], &zero, sizeof(vector int)) == 0) { + fprintf(stderr, "Detected zero from the VSX reg %d\n", i + 12); + return 2; + } + } + + if (memcmp(a, &a[12], 12 * sizeof(vector int))) { + long *p = (long *)a; + fprintf(stderr, "VSX mismatch\n"); + for (i = 0; i < 24; i=i+2) + fprintf(stderr, "%d: 0x%08lx%08lx | 0x%08lx%08lx\n", + i/2 + i%2 + 20, p[i], p[i + 1], p[i + 24], p[i + 25]); + return 1; + } + return 0; +} + +void *preempt_vsx_c(void *p) +{ + int i, j; + long rc; + srand(pthread_self()); + for (i = 0; i < 12; i++) + for (j = 0; j < 4; j++) { + varray[i][j] = rand(); + /* Don't want zero because it hides kernel problems */ + if (varray[i][j] == 0) + j--; + } + rc = preempt_vsx(varray, &threads_starting, &running); + if (rc == 2) + fprintf(stderr, "Caught zeros in VSX compares\n"); + return (void *)rc; +} + +int test_preempt_vsx(void) +{ + int i, rc, threads; + pthread_t *tids; + + threads = sysconf(_SC_NPROCESSORS_ONLN) * THREAD_FACTOR; + tids = malloc(threads * sizeof(pthread_t)); + FAIL_IF(!tids); + + running = true; + threads_starting = threads; + for (i = 0; i < threads; i++) { + rc = pthread_create(&tids[i], NULL, preempt_vsx_c, NULL); + FAIL_IF(rc); + } + + setbuf(stdout, NULL); + /* Not really nessesary but nice to wait for every thread to start */ + printf("\tWaiting for %d workers to start...", threads_starting); + while(threads_starting) + asm volatile("": : :"memory"); + printf("done\n"); + + printf("\tWaiting for %d seconds to let some workers get preempted...", PREEMPT_TIME); + sleep(PREEMPT_TIME); + printf("done\n"); + + printf("\tStopping workers..."); + /* + * Working are checking this value every loop. In preempt_vsx 'cmpwi r5,0; bne 2b'. + * r5 will have loaded the value of running. + */ + running = 0; + for (i = 0; i < threads; i++) { + void *rc_p; + pthread_join(tids[i], &rc_p); + + /* + * Harness will say the fail was here, look at why preempt_vsx + * returned + */ + if ((long) rc_p) + printf("oops\n"); + FAIL_IF((long) rc_p); + } + printf("done\n"); + + return 0; +} + +int main(int argc, char *argv[]) +{ + return test_harness(test_preempt_vsx, "vsx_preempt"); +} diff --git a/tools/testing/selftests/powerpc/signal/Makefile b/tools/testing/selftests/powerpc/signal/Makefile new file mode 100644 index 000000000000..f0eef27458e2 --- /dev/null +++ b/tools/testing/selftests/powerpc/signal/Makefile @@ -0,0 +1,13 @@ +TEST_PROGS := signal signal_tm + +all: $(TEST_PROGS) + +$(TEST_PROGS): ../harness.c ../utils.c signal.S + +CFLAGS += -maltivec +signal_tm: CFLAGS += -mhtm + +include ../../lib.mk + +clean: + rm -f $(TEST_PROGS) *.o diff --git a/tools/testing/selftests/powerpc/signal/signal.S b/tools/testing/selftests/powerpc/signal/signal.S new file mode 100644 index 000000000000..7043d521df0a --- /dev/null +++ b/tools/testing/selftests/powerpc/signal/signal.S @@ -0,0 +1,50 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "../basic_asm.h" + +/* long signal_self(pid_t pid, int sig); */ +FUNC_START(signal_self) + li r0,37 /* sys_kill */ + /* r3 already has our pid in it */ + /* r4 already has signal type in it */ + sc + bc 4,3,1f + subfze r3,r3 +1: blr +FUNC_END(signal_self) + +/* long tm_signal_self(pid_t pid, int sig, int *ret); */ +FUNC_START(tm_signal_self) + PUSH_BASIC_STACK(8) + std r5,STACK_FRAME_PARAM(0)(sp) /* ret */ + tbegin. + beq 1f + tsuspend. + li r0,37 /* sys_kill */ + /* r3 already has our pid in it */ + /* r4 already has signal type in it */ + sc + ld r5,STACK_FRAME_PARAM(0)(sp) /* ret */ + bc 4,3,2f + subfze r3,r3 +2: std r3,0(r5) + tabort. 0 + tresume. /* Be nice to some cleanup, jumps back to tbegin then to 1: */ + /* + * Transaction should be proper doomed and we should never get + * here + */ + li r3,1 + POP_BASIC_STACK(8) + blr +1: li r3,0 + POP_BASIC_STACK(8) + blr +FUNC_END(tm_signal_self) diff --git a/tools/testing/selftests/powerpc/signal/signal.c b/tools/testing/selftests/powerpc/signal/signal.c new file mode 100644 index 000000000000..e7dedd28b3c2 --- /dev/null +++ b/tools/testing/selftests/powerpc/signal/signal.c @@ -0,0 +1,111 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * Sending one self a signal should always get delivered. + */ + +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" + +#define MAX_ATTEMPT 500000 +#define TIMEOUT 5 + +extern long signal_self(pid_t pid, int sig); + +static sig_atomic_t signaled; +static sig_atomic_t fail; + +static void signal_handler(int sig) +{ + if (sig == SIGUSR1) + signaled = 1; + else + fail = 1; +} + +static int test_signal() +{ + int i; + struct sigaction act; + pid_t ppid = getpid(); + pid_t pid; + + act.sa_handler = signal_handler; + act.sa_flags = 0; + sigemptyset(&act.sa_mask); + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction SIGUSR1"); + exit(1); + } + if (sigaction(SIGALRM, &act, NULL) < 0) { + perror("sigaction SIGALRM"); + exit(1); + } + + /* Don't do this for MAX_ATTEMPT, its simply too long */ + for(i = 0; i < 1000; i++) { + pid = fork(); + if (pid == -1) { + perror("fork"); + exit(1); + } + if (pid == 0) { + signal_self(ppid, SIGUSR1); + exit(1); + } else { + alarm(0); /* Disable any pending */ + alarm(2); + while (!signaled && !fail) + asm volatile("": : :"memory"); + if (!signaled) { + fprintf(stderr, "Didn't get signal from child\n"); + FAIL_IF(1); /* For the line number */ + } + /* Otherwise we'll loop too fast and fork() will eventually fail */ + waitpid(pid, NULL, 0); + } + } + + for (i = 0; i < MAX_ATTEMPT; i++) { + long rc; + + alarm(0); /* Disable any pending */ + signaled = 0; + alarm(TIMEOUT); + rc = signal_self(ppid, SIGUSR1); + if (rc) { + fprintf(stderr, "(%d) Fail reason: %d rc=0x%lx", + i, fail, rc); + FAIL_IF(1); /* For the line number */ + } + while (!signaled && !fail) + asm volatile("": : :"memory"); + if (!signaled) { + fprintf(stderr, "(%d) Fail reason: %d rc=0x%lx", + i, fail, rc); + FAIL_IF(1); /* For the line number */ + } + } + + return 0; +} + +int main(void) +{ + test_harness_set_timeout(300); + return test_harness(test_signal, "signal"); +} diff --git a/tools/testing/selftests/powerpc/signal/signal_tm.c b/tools/testing/selftests/powerpc/signal/signal_tm.c new file mode 100644 index 000000000000..2e7451a37cc6 --- /dev/null +++ b/tools/testing/selftests/powerpc/signal/signal_tm.c @@ -0,0 +1,110 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * Sending one self a signal should always get delivered. + */ + +#include <errno.h> +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <signal.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" +#include "../tm/tm.h" + +#define MAX_ATTEMPT 500000 +#define TIMEOUT 10 + +extern long tm_signal_self(pid_t pid, int sig, long *ret); + +static sig_atomic_t signaled; +static sig_atomic_t fail; + +static void signal_handler(int sig) +{ + if (tcheck_active()) { + fail = 2; + return; + } + + if (sig == SIGUSR1) + signaled = 1; + else + fail = 1; +} + +static int test_signal_tm() +{ + int i; + struct sigaction act; + + act.sa_handler = signal_handler; + act.sa_flags = 0; + sigemptyset(&act.sa_mask); + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction SIGUSR1"); + exit(1); + } + if (sigaction(SIGALRM, &act, NULL) < 0) { + perror("sigaction SIGALRM"); + exit(1); + } + + SKIP_IF(!have_htm()); + + for (i = 0; i < MAX_ATTEMPT; i++) { + /* + * If anything bad happens in ASM and we fail to set ret + * because *handwave* TM this will cause failure + */ + long ret = 0xdead; + long rc = 0xbeef; + + alarm(0); /* Disable any pending */ + signaled = 0; + alarm(TIMEOUT); + FAIL_IF(tcheck_transactional()); + rc = tm_signal_self(getpid(), SIGUSR1, &ret); + if (ret == 0xdead) + /* + * This basically means the transaction aborted before we + * even got to the suspend... this is crazy but it + * happens. + * Yes this also means we might never make forward + * progress... the alarm() will trip eventually... + */ + continue; + + if (rc || ret) { + /* Ret is actually an errno */ + printf("TEXASR 0x%016lx, TFIAR 0x%016lx\n", + __builtin_get_texasr(), __builtin_get_tfiar()); + fprintf(stderr, "(%d) Fail reason: %d rc=0x%lx ret=0x%lx\n", + i, fail, rc, ret); + FAIL_IF(ret); + } + while(!signaled && !fail) + asm volatile("": : :"memory"); + if (!signaled) { + fprintf(stderr, "(%d) Fail reason: %d rc=0x%lx ret=0x%lx\n", + i, fail, rc, ret); + FAIL_IF(fail); /* For the line number */ + } + } + + return 0; +} + +int main(void) +{ + return test_harness(test_signal_tm, "signal_tm"); +} diff --git a/tools/testing/selftests/powerpc/tm/Makefile b/tools/testing/selftests/powerpc/tm/Makefile index 9d301d785d9e..c6c53c82fdd6 100644 --- a/tools/testing/selftests/powerpc/tm/Makefile +++ b/tools/testing/selftests/powerpc/tm/Makefile @@ -1,5 +1,8 @@ +SIGNAL_CONTEXT_CHK_TESTS := tm-signal-context-chk-gpr tm-signal-context-chk-fpu \ + tm-signal-context-chk-vmx tm-signal-context-chk-vsx + TEST_PROGS := tm-resched-dscr tm-syscall tm-signal-msr-resv tm-signal-stack \ - tm-vmxcopy tm-fork tm-tar tm-tmspr tm-exec tm-execed + tm-vmxcopy tm-fork tm-tar tm-tmspr $(SIGNAL_CONTEXT_CHK_TESTS) all: $(TEST_PROGS) @@ -11,6 +14,9 @@ tm-syscall: tm-syscall-asm.S tm-syscall: CFLAGS += -I../../../../../usr/include tm-tmspr: CFLAGS += -pthread +$(SIGNAL_CONTEXT_CHK_TESTS): tm-signal.S +$(SIGNAL_CONTEXT_CHK_TESTS): CFLAGS += -mhtm -m64 -mvsx + include ../../lib.mk clean: diff --git a/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-fpu.c b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-fpu.c new file mode 100644 index 000000000000..c760debbd5ad --- /dev/null +++ b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-fpu.c @@ -0,0 +1,92 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * + * Test the kernel's signal frame code. + * + * The kernel sets up two sets of ucontexts if the signal was to be + * delivered while the thread was in a transaction. + * Expected behaviour is that the checkpointed state is in the user + * context passed to the signal handler. The speculated state can be + * accessed with the uc_link pointer. + * + * The rationale for this is that if TM unaware code (which linked + * against TM libs) installs a signal handler it will not know of the + * speculative nature of the 'live' registers and may infer the wrong + * thing. + */ + +#include <stdlib.h> +#include <stdio.h> +#include <signal.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" +#include "tm.h" + +#define MAX_ATTEMPT 500000 + +#define NV_FPU_REGS 18 + +long tm_signal_self_context_load(pid_t pid, long *gprs, double *fps, vector int *vms, vector int *vss); + +/* Be sure there are 2x as many as there are NV FPU regs (2x18) */ +static double fps[] = { + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, + -1,-2,-3,-4,-5,-6,-7,-8,-9,-10,-11,-12,-13,-14,-15,-16,-17,-18 +}; + +static sig_atomic_t fail; + +static void signal_usr1(int signum, siginfo_t *info, void *uc) +{ + int i; + ucontext_t *ucp = uc; + ucontext_t *tm_ucp = ucp->uc_link; + + for (i = 0; i < NV_FPU_REGS && !fail; i++) { + fail = (ucp->uc_mcontext.fp_regs[i + 14] != fps[i]); + fail |= (tm_ucp->uc_mcontext.fp_regs[i + 14] != fps[i + NV_FPU_REGS]); + if (fail) + printf("Failed on %d FP %g or %g\n", i, ucp->uc_mcontext.fp_regs[i + 14], tm_ucp->uc_mcontext.fp_regs[i + 14]); + } +} + +static int tm_signal_context_chk_fpu() +{ + struct sigaction act; + int i; + long rc; + pid_t pid = getpid(); + + SKIP_IF(!have_htm()); + + act.sa_sigaction = signal_usr1; + sigemptyset(&act.sa_mask); + act.sa_flags = SA_SIGINFO; + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction sigusr1"); + exit(1); + } + + i = 0; + while (i < MAX_ATTEMPT && !fail) { + rc = tm_signal_self_context_load(pid, NULL, fps, NULL, NULL); + FAIL_IF(rc != pid); + i++; + } + + return fail; +} + +int main(void) +{ + return test_harness(tm_signal_context_chk_fpu, "tm_signal_context_chk_fpu"); +} diff --git a/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-gpr.c b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-gpr.c new file mode 100644 index 000000000000..df91330a08ef --- /dev/null +++ b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-gpr.c @@ -0,0 +1,90 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * + * Test the kernel's signal frame code. + * + * The kernel sets up two sets of ucontexts if the signal was to be + * delivered while the thread was in a transaction. + * Expected behaviour is that the checkpointed state is in the user + * context passed to the signal handler. The speculated state can be + * accessed with the uc_link pointer. + * + * The rationale for this is that if TM unaware code (which linked + * against TM libs) installs a signal handler it will not know of the + * speculative nature of the 'live' registers and may infer the wrong + * thing. + */ + +#include <stdlib.h> +#include <stdio.h> +#include <signal.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" +#include "tm.h" + +#define MAX_ATTEMPT 500000 + +#define NV_GPR_REGS 18 + +long tm_signal_self_context_load(pid_t pid, long *gprs, double *fps, vector int *vms, vector int *vss); + +static sig_atomic_t fail; + +static long gps[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, + -1,-2,-3,-4,-5,-6,-7,-8,-9,-10,-11,-12,-13,-14,-15,-16,-17,-18}; + +static void signal_usr1(int signum, siginfo_t *info, void *uc) +{ + int i; + ucontext_t *ucp = uc; + ucontext_t *tm_ucp = ucp->uc_link; + + for (i = 0; i < NV_GPR_REGS && !fail; i++) { + fail = (ucp->uc_mcontext.gp_regs[i + 14] != gps[i]); + fail |= (tm_ucp->uc_mcontext.gp_regs[i + 14] != gps[i + NV_GPR_REGS]); + if (fail) + printf("Failed on %d GPR %lu or %lu\n", i, + ucp->uc_mcontext.gp_regs[i + 14], tm_ucp->uc_mcontext.gp_regs[i + 14]); + } +} + +static int tm_signal_context_chk_gpr() +{ + struct sigaction act; + int i; + long rc; + pid_t pid = getpid(); + + SKIP_IF(!have_htm()); + + act.sa_sigaction = signal_usr1; + sigemptyset(&act.sa_mask); + act.sa_flags = SA_SIGINFO; + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction sigusr1"); + exit(1); + } + + i = 0; + while (i < MAX_ATTEMPT && !fail) { + rc = tm_signal_self_context_load(pid, gps, NULL, NULL, NULL); + FAIL_IF(rc != pid); + i++; + } + + return fail; +} + +int main(void) +{ + return test_harness(tm_signal_context_chk_gpr, "tm_signal_context_chk_gpr"); +} diff --git a/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vmx.c b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vmx.c new file mode 100644 index 000000000000..f0ee55fd5185 --- /dev/null +++ b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vmx.c @@ -0,0 +1,110 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * + * Test the kernel's signal frame code. + * + * The kernel sets up two sets of ucontexts if the signal was to be + * delivered while the thread was in a transaction. + * Expected behaviour is that the checkpointed state is in the user + * context passed to the signal handler. The speculated state can be + * accessed with the uc_link pointer. + * + * The rationale for this is that if TM unaware code (which linked + * against TM libs) installs a signal handler it will not know of the + * speculative nature of the 'live' registers and may infer the wrong + * thing. + */ + +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <signal.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" +#include "tm.h" + +#define MAX_ATTEMPT 500000 + +#define NV_VMX_REGS 12 + +long tm_signal_self_context_load(pid_t pid, long *gprs, double *fps, vector int *vms, vector int *vss); + +static sig_atomic_t fail; + +vector int vms[] = { + {1, 2, 3, 4 },{5, 6, 7, 8 },{9, 10,11,12}, + {13,14,15,16},{17,18,19,20},{21,22,23,24}, + {25,26,27,28},{29,30,31,32},{33,34,35,36}, + {37,38,39,40},{41,42,43,44},{45,46,47,48}, + {-1, -2, -3, -4}, {-5, -6, -7, -8}, {-9, -10,-11,-12}, + {-13,-14,-15,-16},{-17,-18,-19,-20},{-21,-22,-23,-24}, + {-25,-26,-27,-28},{-29,-30,-31,-32},{-33,-34,-35,-36}, + {-37,-38,-39,-40},{-41,-42,-43,-44},{-45,-46,-47,-48} +}; + +static void signal_usr1(int signum, siginfo_t *info, void *uc) +{ + int i; + ucontext_t *ucp = uc; + ucontext_t *tm_ucp = ucp->uc_link; + + for (i = 0; i < NV_VMX_REGS && !fail; i++) { + fail = memcmp(ucp->uc_mcontext.v_regs->vrregs[i + 20], + &vms[i], sizeof(vector int)); + fail |= memcmp(tm_ucp->uc_mcontext.v_regs->vrregs[i + 20], + &vms[i + NV_VMX_REGS], sizeof (vector int)); + + if (fail) { + int j; + + fprintf(stderr, "Failed on %d vmx 0x", i); + for (j = 0; j < 4; j++) + fprintf(stderr, "%04x", ucp->uc_mcontext.v_regs->vrregs[i + 20][j]); + fprintf(stderr, " vs 0x"); + for (j = 0 ; j < 4; j++) + fprintf(stderr, "%04x", tm_ucp->uc_mcontext.v_regs->vrregs[i + 20][j]); + fprintf(stderr, "\n"); + } + } +} + +static int tm_signal_context_chk() +{ + struct sigaction act; + int i; + long rc; + pid_t pid = getpid(); + + SKIP_IF(!have_htm()); + + act.sa_sigaction = signal_usr1; + sigemptyset(&act.sa_mask); + act.sa_flags = SA_SIGINFO; + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction sigusr1"); + exit(1); + } + + i = 0; + while (i < MAX_ATTEMPT && !fail) { + rc = tm_signal_self_context_load(pid, NULL, NULL, vms, NULL); + FAIL_IF(rc != pid); + i++; + } + + return fail; +} + +int main(void) +{ + return test_harness(tm_signal_context_chk, "tm_signal_context_chk_vmx"); +} diff --git a/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vsx.c b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vsx.c new file mode 100644 index 000000000000..b99c3d835957 --- /dev/null +++ b/tools/testing/selftests/powerpc/tm/tm-signal-context-chk-vsx.c @@ -0,0 +1,125 @@ +/* + * Copyright 2016, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * + * Test the kernel's signal frame code. + * + * The kernel sets up two sets of ucontexts if the signal was to be + * delivered while the thread was in a transaction. + * Expected behaviour is that the checkpointed state is in the user + * context passed to the signal handler. The speculated state can be + * accessed with the uc_link pointer. + * + * The rationale for this is that if TM unaware code (which linked + * against TM libs) installs a signal handler it will not know of the + * speculative nature of the 'live' registers and may infer the wrong + * thing. + */ + +#include <stdlib.h> +#include <stdio.h> +#include <string.h> +#include <signal.h> +#include <unistd.h> + +#include <altivec.h> + +#include "utils.h" +#include "tm.h" + +#define MAX_ATTEMPT 500000 + +#define NV_VSX_REGS 12 + +long tm_signal_self_context_load(pid_t pid, long *gprs, double *fps, vector int *vms, vector int *vss); + +static sig_atomic_t fail; + +vector int vss[] = { + {1, 2, 3, 4 },{5, 6, 7, 8 },{9, 10,11,12}, + {13,14,15,16},{17,18,19,20},{21,22,23,24}, + {25,26,27,28},{29,30,31,32},{33,34,35,36}, + {37,38,39,40},{41,42,43,44},{45,46,47,48}, + {-1, -2, -3, -4 },{-5, -6, -7, -8 },{-9, -10,-11,-12}, + {-13,-14,-15,-16},{-17,-18,-19,-20},{-21,-22,-23,-24}, + {-25,-26,-27,-28},{-29,-30,-31,-32},{-33,-34,-35,-36}, + {-37,-38,-39,-40},{-41,-42,-43,-44},{-45,-46,-47,-48} +}; + +static void signal_usr1(int signum, siginfo_t *info, void *uc) +{ + int i; + uint8_t vsc[sizeof(vector int)]; + uint8_t vst[sizeof(vector int)]; + ucontext_t *ucp = uc; + ucontext_t *tm_ucp = ucp->uc_link; + + /* + * The other half of the VSX regs will be after v_regs. + * + * In short, vmx_reserve array holds everything. v_regs is a 16 + * byte aligned pointer at the start of vmx_reserve (vmx_reserve + * may or may not be 16 aligned) where the v_regs structure exists. + * (half of) The VSX regsters are directly after v_regs so the + * easiest way to find them below. + */ + long *vsx_ptr = (long *)(ucp->uc_mcontext.v_regs + 1); + long *tm_vsx_ptr = (long *)(tm_ucp->uc_mcontext.v_regs + 1); + for (i = 0; i < NV_VSX_REGS && !fail; i++) { + memcpy(vsc, &ucp->uc_mcontext.fp_regs[i + 20], 8); + memcpy(vsc + 8, &vsx_ptr[20 + i], 8); + fail = memcmp(vsc, &vss[i], sizeof(vector int)); + memcpy(vst, &tm_ucp->uc_mcontext.fp_regs[i + 20], 8); + memcpy(vst + 8, &tm_vsx_ptr[20 + i], 8); + fail |= memcmp(vst, &vss[i + NV_VSX_REGS], sizeof(vector int)); + + if (fail) { + int j; + + fprintf(stderr, "Failed on %d vsx 0x", i); + for (j = 0; j < 16; j++) + fprintf(stderr, "%02x", vsc[j]); + fprintf(stderr, " vs 0x"); + for (j = 0; j < 16; j++) + fprintf(stderr, "%02x", vst[j]); + fprintf(stderr, "\n"); + } + } +} + +static int tm_signal_context_chk() +{ + struct sigaction act; + int i; + long rc; + pid_t pid = getpid(); + + SKIP_IF(!have_htm()); + + act.sa_sigaction = signal_usr1; + sigemptyset(&act.sa_mask); + act.sa_flags = SA_SIGINFO; + if (sigaction(SIGUSR1, &act, NULL) < 0) { + perror("sigaction sigusr1"); + exit(1); + } + + i = 0; + while (i < MAX_ATTEMPT && !fail) { + rc = tm_signal_self_context_load(pid, NULL, NULL, NULL, vss); + FAIL_IF(rc != pid); + i++; + } + + return fail; +} + +int main(void) +{ + return test_harness(tm_signal_context_chk, "tm_signal_context_chk_vsx"); +} diff --git a/tools/testing/selftests/powerpc/tm/tm-signal.S b/tools/testing/selftests/powerpc/tm/tm-signal.S new file mode 100644 index 000000000000..4e13e8b3a96f --- /dev/null +++ b/tools/testing/selftests/powerpc/tm/tm-signal.S @@ -0,0 +1,114 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "../basic_asm.h" +#include "../gpr_asm.h" +#include "../fpu_asm.h" +#include "../vmx_asm.h" +#include "../vsx_asm.h" + +/* + * Large caveat here being that the caller cannot expect the + * signal to always be sent! The hardware can (AND WILL!) abort + * the transaction between the tbegin and the tsuspend (however + * unlikely it seems or infrequently it actually happens). + * You have been warned. + */ +/* long tm_signal_self(pid_t pid, long *gprs, double *fps, vector *vms, vector *vss); */ +FUNC_START(tm_signal_self_context_load) + PUSH_BASIC_STACK(512) + /* + * Don't strictly need to save and restore as it depends on if + * we're going to use them, however this reduces messy logic + */ + PUSH_VMX(STACK_FRAME_LOCAL(5,0),r8) + PUSH_FPU(512) + PUSH_NVREGS_BELOW_FPU(512) + std r3, STACK_FRAME_PARAM(0)(sp) /* pid */ + std r4, STACK_FRAME_PARAM(1)(sp) /* gps */ + std r5, STACK_FRAME_PARAM(2)(sp) /* fps */ + std r6, STACK_FRAME_PARAM(3)(sp) /* vms */ + std r7, STACK_FRAME_PARAM(4)(sp) /* vss */ + + ld r3, STACK_FRAME_PARAM(1)(sp) + cmpdi r3, 0 + beq skip_gpr_lc + bl load_gpr +skip_gpr_lc: + ld r3, STACK_FRAME_PARAM(2)(sp) + cmpdi r3, 0 + beq skip_fpu_lc + bl load_fpu +skip_fpu_lc: + ld r3, STACK_FRAME_PARAM(3)(sp) + cmpdi r3, 0 + beq skip_vmx_lc + bl load_vmx +skip_vmx_lc: + ld r3, STACK_FRAME_PARAM(4)(sp) + cmpdi r3, 0 + beq skip_vsx_lc + bl load_vsx +skip_vsx_lc: + /* + * Set r3 (return value) before tbegin. Use the pid as a known + * 'all good' return value, zero is used to indicate a non-doomed + * transaction. + */ + ld r3, STACK_FRAME_PARAM(0)(sp) + tbegin. + beq 1f + tsuspend. /* Can't enter a syscall transactionally */ + ld r3, STACK_FRAME_PARAM(1)(sp) + cmpdi r3, 0 + beq skip_gpr_lt + /* Get the second half of the array */ + addi r3, r3, 8 * 18 + bl load_gpr +skip_gpr_lt: + ld r3, STACK_FRAME_PARAM(2)(sp) + cmpdi r3, 0 + beq skip_fpu_lt + /* Get the second half of the array */ + addi r3, r3, 8 * 18 + bl load_fpu +skip_fpu_lt: + ld r3, STACK_FRAME_PARAM(3)(sp) + cmpdi r3, 0 + beq skip_vmx_lt + /* Get the second half of the array */ + addi r3, r3, 16 * 12 + bl load_vmx +skip_vmx_lt: + ld r3, STACK_FRAME_PARAM(4)(sp) + cmpdi r3, 0 + beq skip_vsx_lt + /* Get the second half of the array */ + addi r3, r3, 16 * 12 + bl load_vsx +skip_vsx_lt: + li r0, 37 /* sys_kill */ + ld r3, STACK_FRAME_PARAM(0)(sp) /* pid */ + li r4, 10 /* SIGUSR1 */ + sc /* Taking the signal will doom the transaction */ + tabort. 0 + tresume. /* Be super sure we abort */ + /* + * This will cause us to resume doomed transaction and cause + * hardware to cleanup, we'll end up at 1: anything between + * tresume. and 1: shouldn't ever run. + */ + li r3, 0 + 1: + POP_VMX(STACK_FRAME_LOCAL(5,0),r4) + POP_FPU(512) + POP_NVREGS_BELOW_FPU(512) + POP_BASIC_STACK(512) + blr +FUNC_END(tm_signal_self_context_load) diff --git a/tools/testing/selftests/powerpc/tm/tm.h b/tools/testing/selftests/powerpc/tm/tm.h index 60318bad7d7a..2c8da74304e7 100644 --- a/tools/testing/selftests/powerpc/tm/tm.h +++ b/tools/testing/selftests/powerpc/tm/tm.h @@ -52,4 +52,31 @@ static inline bool failure_is_nesting(void) return (__builtin_get_texasru() & 0x400000); } +static inline int tcheck(void) +{ + long cr; + asm volatile ("tcheck 0" : "=r"(cr) : : "cr0"); + return (cr >> 28) & 4; +} + +static inline bool tcheck_doomed(void) +{ + return tcheck() & 8; +} + +static inline bool tcheck_active(void) +{ + return tcheck() & 4; +} + +static inline bool tcheck_suspended(void) +{ + return tcheck() & 2; +} + +static inline bool tcheck_transactional(void) +{ + return tcheck() & 6; +} + #endif /* _SELFTESTS_POWERPC_TM_TM_H */ diff --git a/tools/testing/selftests/powerpc/utils.h b/tools/testing/selftests/powerpc/utils.h index fbd33e52ef8f..53405e8a52ab 100644 --- a/tools/testing/selftests/powerpc/utils.h +++ b/tools/testing/selftests/powerpc/utils.h @@ -22,7 +22,7 @@ typedef uint32_t u32; typedef uint16_t u16; typedef uint8_t u8; - +void test_harness_set_timeout(uint64_t time); int test_harness(int (test_function)(void), char *name); extern void *get_auxv_entry(int type); int pick_online_cpu(void); @@ -32,10 +32,17 @@ static inline bool have_hwcap(unsigned long ftr) return ((unsigned long)get_auxv_entry(AT_HWCAP) & ftr) == ftr; } +#ifdef AT_HWCAP2 static inline bool have_hwcap2(unsigned long ftr2) { return ((unsigned long)get_auxv_entry(AT_HWCAP2) & ftr2) == ftr2; } +#else +static inline bool have_hwcap2(unsigned long ftr2) +{ + return false; +} +#endif /* Yes, this is evil */ #define FAIL_IF(x) \ diff --git a/tools/testing/selftests/powerpc/vmx_asm.h b/tools/testing/selftests/powerpc/vmx_asm.h new file mode 100644 index 000000000000..2eaaeca9cf1d --- /dev/null +++ b/tools/testing/selftests/powerpc/vmx_asm.h @@ -0,0 +1,96 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "basic_asm.h" + +/* POS MUST BE 16 ALIGNED! */ +#define PUSH_VMX(pos,reg) \ + li reg,pos; \ + stvx v20,reg,%r1; \ + addi reg,reg,16; \ + stvx v21,reg,%r1; \ + addi reg,reg,16; \ + stvx v22,reg,%r1; \ + addi reg,reg,16; \ + stvx v23,reg,%r1; \ + addi reg,reg,16; \ + stvx v24,reg,%r1; \ + addi reg,reg,16; \ + stvx v25,reg,%r1; \ + addi reg,reg,16; \ + stvx v26,reg,%r1; \ + addi reg,reg,16; \ + stvx v27,reg,%r1; \ + addi reg,reg,16; \ + stvx v28,reg,%r1; \ + addi reg,reg,16; \ + stvx v29,reg,%r1; \ + addi reg,reg,16; \ + stvx v30,reg,%r1; \ + addi reg,reg,16; \ + stvx v31,reg,%r1; + +/* POS MUST BE 16 ALIGNED! */ +#define POP_VMX(pos,reg) \ + li reg,pos; \ + lvx v20,reg,%r1; \ + addi reg,reg,16; \ + lvx v21,reg,%r1; \ + addi reg,reg,16; \ + lvx v22,reg,%r1; \ + addi reg,reg,16; \ + lvx v23,reg,%r1; \ + addi reg,reg,16; \ + lvx v24,reg,%r1; \ + addi reg,reg,16; \ + lvx v25,reg,%r1; \ + addi reg,reg,16; \ + lvx v26,reg,%r1; \ + addi reg,reg,16; \ + lvx v27,reg,%r1; \ + addi reg,reg,16; \ + lvx v28,reg,%r1; \ + addi reg,reg,16; \ + lvx v29,reg,%r1; \ + addi reg,reg,16; \ + lvx v30,reg,%r1; \ + addi reg,reg,16; \ + lvx v31,reg,%r1; + +/* + * Careful this will 'clobber' vmx (by design) + * Don't call this from C + */ +FUNC_START(load_vmx) + li r5,0 + lvx v20,r5,r3 + addi r5,r5,16 + lvx v21,r5,r3 + addi r5,r5,16 + lvx v22,r5,r3 + addi r5,r5,16 + lvx v23,r5,r3 + addi r5,r5,16 + lvx v24,r5,r3 + addi r5,r5,16 + lvx v25,r5,r3 + addi r5,r5,16 + lvx v26,r5,r3 + addi r5,r5,16 + lvx v27,r5,r3 + addi r5,r5,16 + lvx v28,r5,r3 + addi r5,r5,16 + lvx v29,r5,r3 + addi r5,r5,16 + lvx v30,r5,r3 + addi r5,r5,16 + lvx v31,r5,r3 + blr +FUNC_END(load_vmx) diff --git a/tools/testing/selftests/powerpc/vsx_asm.h b/tools/testing/selftests/powerpc/vsx_asm.h new file mode 100644 index 000000000000..d828bfb6ef2d --- /dev/null +++ b/tools/testing/selftests/powerpc/vsx_asm.h @@ -0,0 +1,71 @@ +/* + * Copyright 2015, Cyril Bur, IBM Corp. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include "basic_asm.h" + +/* + * Careful this will 'clobber' vsx (by design), VSX are always + * volatile though so unlike vmx this isn't so much of an issue + * Still should avoid calling from C + */ +FUNC_START(load_vsx) + li r5,0 + lxvx vs20,r5,r3 + addi r5,r5,16 + lxvx vs21,r5,r3 + addi r5,r5,16 + lxvx vs22,r5,r3 + addi r5,r5,16 + lxvx vs23,r5,r3 + addi r5,r5,16 + lxvx vs24,r5,r3 + addi r5,r5,16 + lxvx vs25,r5,r3 + addi r5,r5,16 + lxvx vs26,r5,r3 + addi r5,r5,16 + lxvx vs27,r5,r3 + addi r5,r5,16 + lxvx vs28,r5,r3 + addi r5,r5,16 + lxvx vs29,r5,r3 + addi r5,r5,16 + lxvx vs30,r5,r3 + addi r5,r5,16 + lxvx vs31,r5,r3 + blr +FUNC_END(load_vsx) + +FUNC_START(store_vsx) + li r5,0 + stxvx vs20,r5,r3 + addi r5,r5,16 + stxvx vs21,r5,r3 + addi r5,r5,16 + stxvx vs22,r5,r3 + addi r5,r5,16 + stxvx vs23,r5,r3 + addi r5,r5,16 + stxvx vs24,r5,r3 + addi r5,r5,16 + stxvx vs25,r5,r3 + addi r5,r5,16 + stxvx vs26,r5,r3 + addi r5,r5,16 + stxvx vs27,r5,r3 + addi r5,r5,16 + stxvx vs28,r5,r3 + addi r5,r5,16 + stxvx vs29,r5,r3 + addi r5,r5,16 + stxvx vs30,r5,r3 + addi r5,r5,16 + stxvx vs31,r5,r3 + blr +FUNC_END(store_vsx) |