diff options
| author | Whi copybara merger <whitechapel-automerger@google.com> | 2021-08-20 14:56:11 +0800 |
|---|---|---|
| committer | Sharad Bagri <sharadbagri@google.com> | 2021-09-01 10:26:30 -0700 |
| commit | 75eaaf7fa677bc39ecc02f174bb4a6f48ab76d84 (patch) | |
| tree | f43302fca181ac600c9dccd00fbb8b2ca17ebdc1 | |
| parent | aa19c3a1321d9596f733003f356c722e9f7ed37f (diff) | |
| download | janeiro-75eaaf7fa677bc39ecc02f174bb4a6f48ab76d84.tar.gz | |
[Copybara Auto Merge] Merge branch 'pro' into android13-gs-pixel-5.10
edgetpu: fix potential KP on device probing
Bug: 197144362
edgetpu: janeiro: disable SSMT temporarily
Bug: 197285064
edgetpu: Reset dump memory after handling dump.
edgetpu: abrolhos: Collect host side debug dump segments.
Bug: 176555699
Bug: 168082238
Bug: 176556808
edgetpu: Sync debug dump contract with FW
edgetpu: restrict printing a couple of kernel and physical addresses
edgetpu: use %# instead of 0x% for hex printks
edgetpu: use __s32 for int fd field of dma-buf map ioctl
edgetpu: fix UAF in edgetpu_device_group_map
Bug: 192641593
edgetpu: hermosa add support for firmware signature checks
Bug: 195776283
edgetpu: add physical address field to coherent memory
Bug: 183621442
edgetpu: fix invalid wait context
Bug: 196340168
edgetpu: janeiro: set up SSMT registers
Bug: 196494143
edgetpu: janeiro: Only poll for blk status after shutdown
Bug: 191869036
edgetpu: add sysfs groups attr
edgetpu: add user_vote for user space request TPU throttling
Bug: 188701596
edgetpu: debugfs mappings improvements
edgetpu: use pin_user_pages, alloc own vma struct pointers array
Bug: 194989197
edgetpu: abrolhos disable ext mbox on client remove
Bug: 195126121
edgetpu: don't fail mbox deactivation on KCI fails
Bug: 195126121
edgetpu: add group id to sysfs clients attr, fix output
Bug: 193484549
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0:
edgetpu: janeiro: fix firmware header mappings
edgetpu: mobile guard bcl inclusion with KCONFIG
edgetpu: janeiro remove obsolete TODO
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0:
edgetpu: use kvmalloc for pages array
edgetpu: janeiro: Stability fix in blk pwr control
edgetpu: add support for multiple interfaces
edgetpu: restrict debugfs statusregs file to non-mobile chipsets
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0: (65 commits)
edgetpu: add support to create coherent/non-coherent mapping
edgetpu: add sysfs attr clients to dump client and wakelock state
edgetpu: janeiro: fix device cleanup sequence
edgetpu: add support to track device removal
edgetpu: PM log clients holding TPU wakelocks at suspend reject time
edgetpu: add list of clients per device
edgetpu: rename group client list fields and iterators
edgetpu: abrolhos remove obsolete DT binding
edgetpu: cleanup host DMA direction modification
edgetpu: unittests: Add a firmware crash rkci test
edgetpu: unittests: Add a reverse kci test
edgetpu: hermosa smmu remove TODOs from comments for fw-driven SSID
edgetpu: janeiro bcl port tpu clock divider ratio
edgetpu: unittests: add mobile-platform tests
edgetpu: fix kernel paging error in edgetpu_mmu_attach_domain
edgetpu: janeiro: use DT property to configure shareability
edgetpu: remove remaining references to emulators
edgetpu: hermosa remove obsolete TODO for FPGA USR training
edgetpu: add "is a mobile style device" feature flag
edgetpu: organize objects into mobile vs. mcp groupings
...
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0:
edgetpu: janeiro: fix mailbox offset calculation
edgetpu: unittests: add get_fatal_errors ioctl tests
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0: (62 commits)
edgetpu: unittests: add iremap-pool tests
edgetpu: detach mailbox when activation failed
edgetpu: reduce IOMMU fault reports severity
edgetpu: unittest handle enhanced open device KCI
edgetpu: gem5: Increasing the mask of gsa
edgetpu: unittests: upgrade to KUnit 5.10
edgetpu: unittests: fix tests with disabled IOMMU
edgetpu: allow buffer unmapping on errored groups
edgetpu: abrolhos return actual error for throttling kci
edgetpu: handle job lockup notification from firmware
edgetpu: add API to send fatal error notifications to a specific group
edgetpu: add fatal error event for firmware-detected job timeout
edgetpu: Improve the check in edgetpu_thermal_* functions
edgetpu: Modify the log function due to thermal suspended
edgetpu: fail wakelock acquiring if suspended
edgetpu: ignore offset arg in edgetpu_map_dmabuf
edgetpu: hermosa assume single die when config not set
edgetpu: unittests: add thermal test
edgetpu: abrolhos: hook exynos acpm functions
edgetpu: unittests: add helper of device tree
...
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0: (23 commits)
edgetpu: remove redundant args
edgetpu: janeiro: add power management
edgetpu: janeiro scrub references to TPU CPU model
edgetpu: abrolhos scrub more TPU CPU references
edgetpu: remove some references to codenames and hardware details
edgetpu: remove some details from usage-stats comments and symbols
edgetpu: fix typo in debug dump header
Mock components to build TPU driver
edgetpu: log error when firmware load failed
edgetpu: google: use default domain when AUX disabled
edgetpu: fix edgetpu_mmu_alloc_domain memory leak
edgetpu: remove dependency on iommu_group_id
edgetpu: janeiro: use NS iommu mapping for f/w
edgetpu: fix watchdog job cancel ordering
edgetpu: don't check mailbox_detachable on fs_release
edgetpu: add dev_inaccessible field to struct group
edgetpu: hermosa remove irqreturn header from smmu
edgetpu: remove chip names from mmu.h
edgetpu: move mailbox disable out from mbox remove
edgetpu: export mailbox enable and disable functions
...
Merge remote-tracking branch 'origin/darwinn-2.0' into pro
* origin/darwinn-2.0: (82 commits)
edgetpu: janeiro: parse fw header to create IOMMU mapping
edgetpu: dmabuf map add debug logs for diagnosing errors
edgetpu: make various usage metrics group writeable
edgetpu: hermosa remove two obsolete TODOs
kokoro: checkpatch ignore constant comparison
edgetpu: unittest: remove obsolete bug
edgetpu: set cacheability according to device
edgetpu: add offset back to dmabuf mapping
edgetpu: refactor error handling
edgetpu: reset thread stats on write
edgetpu: abrolhos remove parsing csr-iova nodes
edgetpu: remove mapping CSRs on mmu setup
edgetpu: add corrupted log queue msg back
edgetpu: deprecate offset in edgetpu_map_dmabuf_ioctl
edgetpu: fix use-after-free Read in edgetpu_chip_acquire_ext_mailbox
edgetpu: map dmabuf ignore size argument
edgetpu: hermosa fix UAF in edgetpu_device_dram_getfd
edgetpu: abrolhos fix thermal notify null deref
edgetpu: fix UAF write on edgetpu_sync_fence_create
edgetpu: add firmware_is_loading utility
...
Remove hermosa/abrolhos references on pro branch
Also remove unittests and fix Makefile/Kbuild for
janeiro builds
GitOrigin-RevId: 1805ef57ed473ccdedb44145fa66396f046d984a
Change-Id: I1bb5649c8f367b7762a774d6f310144195a93c28
37 files changed, 599 insertions, 679 deletions
diff --git a/drivers/edgetpu/Kbuild b/drivers/edgetpu/Kbuild index 170a69e..88d8163 100644 --- a/drivers/edgetpu/Kbuild +++ b/drivers/edgetpu/Kbuild @@ -14,6 +14,7 @@ edgetpu-objs := edgetpu-mailbox.o edgetpu-kci.o edgetpu-telemetry.o edgetpu-mapp edgetpu-mobile-objs := edgetpu-mobile-firmware.o + janeiro-y := janeiro-device.o janeiro-device-group.o janeiro-fs.o janeiro-core.o janeiro-platform.o janeiro-firmware.o janeiro-pm.o janeiro-debug-dump.o janeiro-usage-stats.o janeiro-iommu.o janeiro-wakelock.o $(edgetpu-mobile-objs) $(edgetpu-objs) CFLAGS_janeiro-fs.o := -DCONFIG_JANEIRO=1 diff --git a/drivers/edgetpu/Kconfig b/drivers/edgetpu/Kconfig index 0436086..4970c54 100644 --- a/drivers/edgetpu/Kconfig +++ b/drivers/edgetpu/Kconfig @@ -10,17 +10,6 @@ config EDGETPU_FRAMEWORK help This framework supports darwinn-2.0 devices. -config ABROLHOS - tristate "Abrolhos ML accelerator direct host control driver" - depends on EDGETPU_FRAMEWORK - select PM - help - This driver supports the Abrolhos device. Say Y if you want to - include this driver in the kernel. - - To compile this driver as a module, choose M here. The module - will be called "abrolhos". - config JANEIRO tristate "Janeiro ML accelerator device driver" depends on EDGETPU_FRAMEWORK @@ -32,24 +21,6 @@ config JANEIRO To compile this driver as a module, choose M here. The module will be called "janeiro". -config HERMOSA - tristate "Hermosa edgetpu ML accelerator direct host control driver" - depends on EDGETPU_FRAMEWORK - select INTERVAL_TREE if EDGETPU_DEVICE_DRAM - help - This driver supports the Hermosa device. Say Y if you want to - include this driver in the kernel. - - To compile this driver as a module, choose M here. The module - will be called "hermosa". - -config EDGETPU_DEVICE_DRAM - bool "EdgeTPU Device DRAM support" - depends on HERMOSA - default y - help - Say Y if the target device has the embedded device DRAM. - config EDGETPU_EXTERNAL_WRAPPER_CLASS bool "EdgeTPU add external wrapper class" depends on EDGETPU_FRAMEWORK @@ -72,6 +43,4 @@ config EDGETPU_TELEMETRY_TRACE It's fine to have this enabled even the firmware doesn't send tracing events. -source "drivers/edgetpu/unittests/Kconfig" - endmenu diff --git a/drivers/edgetpu/Makefile b/drivers/edgetpu/Makefile index caf01c0..3bbfaaa 100644 --- a/drivers/edgetpu/Makefile +++ b/drivers/edgetpu/Makefile @@ -20,11 +20,14 @@ edgetpu-objs := edgetpu-async.o edgetpu-dmabuf.o edgetpu-iremap-pool.o \ edgetpu-mobile-objs := edgetpu-mobile-firmware.o + janeiro-objs := janeiro-core.o janeiro-debug-dump.o janeiro-device-group.o \ janeiro-device.o janeiro-firmware.o janeiro-fs.o \ janeiro-iommu.o janeiro-platform.o janeiro-pm.o \ janeiro-usage-stats.o janeiro-wakelock.o \ $(edgetpu-objs) $(edgetpu-mobile-objs) +KBUILD_OPTIONS += CONFIG_JANEIRO=m + modules modules_install clean: $(MAKE) -C $(KERNEL_SRC) M=$(M) W=1 $(KBUILD_OPTIONS) $(@) diff --git a/drivers/edgetpu/edgetpu-config.h b/drivers/edgetpu/edgetpu-config.h index 5a13adb..9306fac 100644 --- a/drivers/edgetpu/edgetpu-config.h +++ b/drivers/edgetpu/edgetpu-config.h @@ -8,28 +8,10 @@ #ifndef __EDGETPU_CONFIG_H__ #define __EDGETPU_CONFIG_H__ -#ifdef CONFIG_HERMOSA - -#include "hermosa/config.h" - -#else /* !CONFIG_HERMOSA */ - #ifdef CONFIG_JANEIRO #include "janeiro/config.h" - -#else - -#ifndef CONFIG_ABROLHOS -#define CONFIG_ABROLHOS -#warning "Building default chipset abrolhos" -#endif - -#include "abrolhos/config.h" - #endif /* CONFIG_JANEIRO */ -#endif /* CONFIG_HERMOSA */ - #define EDGETPU_DEFAULT_FIRMWARE_NAME "google/edgetpu-" DRIVER_NAME ".fw" #define EDGETPU_TEST_FIRMWARE_NAME "google/edgetpu-" DRIVER_NAME "-test.fw" diff --git a/drivers/edgetpu/edgetpu-core.c b/drivers/edgetpu/edgetpu-core.c index 3115f17..20e3372 100644 --- a/drivers/edgetpu/edgetpu-core.c +++ b/drivers/edgetpu/edgetpu-core.c @@ -224,12 +224,12 @@ int edgetpu_mmap(struct edgetpu_client *client, struct vm_area_struct *vma) if (vma->vm_start & ~PAGE_MASK) { etdev_dbg(client->etdev, - "Base address not page-aligned: 0x%lx\n", + "Base address not page-aligned: %#lx\n", vma->vm_start); return -EINVAL; } - etdev_dbg(client->etdev, "%s: mmap pgoff = %lX\n", __func__, + etdev_dbg(client->etdev, "%s: mmap pgoff = %#lX\n", __func__, vma->vm_pgoff); type = mmap_vma_type(vma->vm_pgoff); @@ -545,6 +545,8 @@ void edgetpu_client_remove(struct edgetpu_client *client) */ if (client->group) edgetpu_device_group_leave(client); + /* invoke chip-dependent removal handler before releasing resources */ + edgetpu_chip_client_remove(client); edgetpu_wakelock_free(client->wakelock); /* * It should be impossible to access client->wakelock after this cleanup @@ -561,7 +563,6 @@ void edgetpu_client_remove(struct edgetpu_client *client) 1 << perdie_event_id_to_num(EDGETPU_PERDIE_EVENT_TRACES_AVAILABLE)) edgetpu_telemetry_unset_event(etdev, EDGETPU_TELEMETRY_TRACE); - edgetpu_chip_client_remove(client); edgetpu_client_put(client); } diff --git a/drivers/edgetpu/edgetpu-debug-dump.c b/drivers/edgetpu/edgetpu-debug-dump.c index d8ccde9..3836e85 100644 --- a/drivers/edgetpu/edgetpu-debug-dump.c +++ b/drivers/edgetpu/edgetpu-debug-dump.c @@ -9,15 +9,10 @@ #include "edgetpu-config.h" #include "edgetpu-debug-dump.h" +#include "edgetpu-device-group.h" #include "edgetpu-iremap-pool.h" #include "edgetpu-kci.h" -static inline u64 word_align_offset(u64 offset) -{ - return offset/sizeof(u64) + - (((offset % sizeof(u64)) == 0) ? 0 : 1); -} - int edgetpu_get_debug_dump(struct edgetpu_dev *etdev, u64 type) { int ret; @@ -43,52 +38,50 @@ int edgetpu_get_debug_dump(struct edgetpu_dev *etdev, u64 type) return ret; } +static void edgetpu_reset_debug_dump(struct edgetpu_dev *etdev) +{ + memset(etdev->debug_dump_mem.vaddr, 0, etdev->debug_dump_mem.size); +} + static void edgetpu_debug_dump_work(struct work_struct *work) { struct edgetpu_dev *etdev; struct edgetpu_debug_dump_setup *dump_setup; struct edgetpu_debug_dump *debug_dump; int ret; - u64 offset, dump_reason; + u64 dump_reason; etdev = container_of(work, struct edgetpu_dev, debug_dump_work); dump_setup = (struct edgetpu_debug_dump_setup *)etdev->debug_dump_mem.vaddr; - offset = sizeof(struct edgetpu_debug_dump_setup); - debug_dump = (struct edgetpu_debug_dump *)((u64 *)dump_setup + - word_align_offset(offset)); + debug_dump = (struct edgetpu_debug_dump *)(dump_setup + 1); if (!etdev->debug_dump_handlers) { - etdev_err(etdev, - "Failed to generate coredump as handler is NULL"); - goto debug_dump_work_done; + etdev_err(etdev, "Failed to generate coredump as handler is NULL"); + edgetpu_reset_debug_dump(etdev); + return; } - dump_reason = dump_setup->dump_req_reason; - if (dump_reason >= DUMP_REQ_REASON_NUM || + dump_reason = debug_dump->dump_reason; + if (dump_reason >= DUMP_REASON_NUM || !etdev->debug_dump_handlers[dump_reason]) { etdev_err(etdev, - "Failed to generate coredump as handler is NULL for dump request reason: 0x%llx", + "Failed to generate coredump as handler is NULL for dump request reason: %#llx", dump_reason); - goto debug_dump_work_done; + edgetpu_reset_debug_dump(etdev); + return; } - ret = etdev->debug_dump_handlers[dump_reason] - ((void *)etdev, (void *)dump_setup); - if (ret) { + ret = etdev->debug_dump_handlers[dump_reason]((void *)etdev, (void *)dump_setup); + if (ret) etdev_err(etdev, "Failed to generate coredump: %d\n", ret); - goto debug_dump_work_done; - } - -debug_dump_work_done: - debug_dump->host_dump_available_to_read = false; + edgetpu_reset_debug_dump(etdev); } void edgetpu_debug_dump_resp_handler(struct edgetpu_dev *etdev) { struct edgetpu_debug_dump_setup *dump_setup; struct edgetpu_debug_dump *debug_dump; - u64 offset; if (!etdev->debug_dump_mem.vaddr) { etdev_err(etdev, "Debug dump memory not allocated"); @@ -96,12 +89,12 @@ void edgetpu_debug_dump_resp_handler(struct edgetpu_dev *etdev) } dump_setup = (struct edgetpu_debug_dump_setup *)etdev->debug_dump_mem.vaddr; - offset = sizeof(struct edgetpu_debug_dump_setup); - debug_dump = (struct edgetpu_debug_dump *)((u64 *)dump_setup + - word_align_offset(offset)); + debug_dump = (struct edgetpu_debug_dump *)(dump_setup + 1); if (!debug_dump->host_dump_available_to_read) return; + debug_dump->host_dump_available_to_read = false; + if (!etdev->debug_dump_work.func) INIT_WORK(&etdev->debug_dump_work, edgetpu_debug_dump_work); diff --git a/drivers/edgetpu/edgetpu-debug-dump.h b/drivers/edgetpu/edgetpu-debug-dump.h index 125ed1a..6a30ef8 100644 --- a/drivers/edgetpu/edgetpu-debug-dump.h +++ b/drivers/edgetpu/edgetpu-debug-dump.h @@ -10,7 +10,7 @@ #include "edgetpu-internal.h" -#define DEBUG_DUMP_HOST_CONTRACT_VERSION 2 +#define DEBUG_DUMP_HOST_CONTRACT_VERSION 3 enum edgetpu_dump_type_bit_position { DUMP_TYPE_CRASH_REASON_BIT = 0, @@ -24,11 +24,16 @@ enum edgetpu_dump_type_bit_position { }; -enum edgetpu_dump_request_reason { - DUMP_REQ_REASON_DEFAULT = 0, - DUMP_REQ_REASON_WDT_TIMEOUT = 1, - DUMP_REQ_REASON_BY_USER = 2, - DUMP_REQ_REASON_NUM = 3 +enum edgetpu_dump_reason { + DUMP_REASON_DEFAULT = 0, + /* Host request reasons */ + DUMP_REASON_REQ_BY_USER = 1, + + /* FW side dump reasons */ + DUMP_REASON_FW_CHECKPOINT = 2, + DUMP_REASON_RECOVERABLE_FAULT = 3, + + DUMP_REASON_NUM = 4 }; struct edgetpu_crash_reason { @@ -51,6 +56,7 @@ struct edgetpu_debug_dump { u64 magic; /* word identifying the beginning of the dump info */ u64 version; /* host-firmware dump info contract version */ u64 host_dump_available_to_read; /* is new info available */ + u64 dump_reason; /* Reason or context for debug dump */ u64 reserved[2]; u64 crash_reason_offset; /* byte offset to crash reason */ u64 crash_reason_size; /* crash reason size */ @@ -64,7 +70,6 @@ struct edgetpu_debug_dump_setup { /* types of dumps requested by host */ u64 type; u64 dump_mem_size; /* total size of memory allocated to dump */ - u64 dump_req_reason; /* debug dump request reason */ u64 reserved[2]; }; diff --git a/drivers/edgetpu/edgetpu-device-group.c b/drivers/edgetpu/edgetpu-device-group.c index d1c0832..6f338be 100644 --- a/drivers/edgetpu/edgetpu-device-group.c +++ b/drivers/edgetpu/edgetpu-device-group.c @@ -41,9 +41,6 @@ #include "edgetpu-p2p-mailbox.h" #endif -#define for_each_list_group_client(c, group) \ - list_for_each_entry(c, &group->clients, list) - #define for_each_list_group_client_safe(c, n, group) \ list_for_each_entry_safe(c, n, &group->clients, list) @@ -143,16 +140,11 @@ static int edgetpu_group_activate(struct edgetpu_device_group *group) static void edgetpu_group_deactivate(struct edgetpu_device_group *group) { u8 mailbox_id; - int ret; if (edgetpu_group_mailbox_detached_locked(group)) return; mailbox_id = edgetpu_group_context_id_locked(group); - ret = edgetpu_mailbox_deactivate(group->etdev, mailbox_id); - if (ret) - etdev_err(group->etdev, "deactivate mailbox for VCID %d failed with %d", - group->vcid, ret); - return; + edgetpu_mailbox_deactivate(group->etdev, mailbox_id); } /* @@ -1064,7 +1056,7 @@ static void edgetpu_unmap_node(struct edgetpu_mapping *map) struct sg_page_iter sg_iter; uint i; - etdev_dbg(group->etdev, "%s: %u: die=%d, iova=0x%llx", __func__, + etdev_dbg(group->etdev, "%s: %u: die=%d, iova=%#llx", __func__, group->workload_id, map->die_index, map->device_address); if (map->device_address) { @@ -1113,7 +1105,7 @@ static void edgetpu_host_map_show(struct edgetpu_mapping *map, seq_puts(s, " mirrored: "); else seq_printf(s, " die %u: ", map->die_index); - seq_printf(s, "0x%llx %lu %s 0x%llx %pap %pad\n", + seq_printf(s, "%#llx %lu %s %#llx %pap %pad\n", map->device_address + cur_offset, DIV_ROUND_UP(sg_dma_len(sg), PAGE_SIZE), edgetpu_dma_dir_rw_s(map->dir), @@ -1123,6 +1115,12 @@ static void edgetpu_host_map_show(struct edgetpu_mapping *map, } } +size_t edgetpu_group_mappings_total_size(struct edgetpu_device_group *group) +{ + return edgetpu_mappings_total_size(&group->host_mappings) + + edgetpu_mappings_total_size(&group->dmabuf_mappings); +} + /* * Pins the user-space address @arg->host_address and returns the pinned pages. * @pnum_pages is set to the number of pages. @@ -1142,6 +1140,7 @@ static struct page **edgetpu_pin_user_pages(struct edgetpu_device_group *group, int i; int ret; struct vm_area_struct *vma; + struct vm_area_struct **vmas; unsigned int foll_flags = FOLL_LONGTERM | FOLL_WRITE; if (size == 0) @@ -1151,18 +1150,17 @@ static struct page **edgetpu_pin_user_pages(struct edgetpu_device_group *group, if (unlikely((size + offset) / PAGE_SIZE >= UINT_MAX - 1 || size + offset < size)) return ERR_PTR(-ENOMEM); num_pages = DIV_ROUND_UP((size + offset), PAGE_SIZE); - etdev_dbg(etdev, "%s: hostaddr=0x%llx pages=%u", __func__, host_addr, num_pages); + etdev_dbg(etdev, "%s: hostaddr=%#llx pages=%u", __func__, host_addr, num_pages); /* * "num_pages" is decided from user-space arguments, don't show warnings * when facing malicious input. */ pages = kvmalloc((num_pages * sizeof(*pages)), GFP_KERNEL | __GFP_NOWARN); if (!pages) { - etdev_dbg(etdev, "%s: kvmalloc failed (%lu bytes)\n", __func__, - (num_pages * sizeof(*pages))); + etdev_dbg(etdev, "%s: kvmalloc pages failed (%lu bytes)\n", + __func__, (num_pages * sizeof(*pages))); return ERR_PTR(-ENOMEM); } - /* * The host pages might be read-only and could fail if we attempt to pin * it with FOLL_WRITE. @@ -1176,10 +1174,43 @@ static struct page **edgetpu_pin_user_pages(struct edgetpu_device_group *group, *preadonly = false; } + /* Try fast call first, in case it's actually faster. */ ret = pin_user_pages_fast(host_addr & PAGE_MASK, num_pages, foll_flags, pages); + if (ret == num_pages) { + *pnum_pages = num_pages; + return pages; + } + if (ret < 0) { + etdev_dbg(etdev, "pin_user_pages failed %u:%pK-%u: %d", + group->workload_id, (void *)host_addr, num_pages, + ret); + if (ret != -ENOMEM) { + num_pages = 0; + goto error; + } + } + etdev_dbg(etdev, + "pin_user_pages_fast error %u:%pK npages=%u ret=%d", + group->workload_id, (void *)host_addr, num_pages, + ret); + /* Unpin any partial mapping and start over again. */ + for (i = 0; i < ret; i++) + unpin_user_page(pages[i]); + + /* Allocate our own vmas array non-contiguous. */ + vmas = kvmalloc((num_pages * sizeof(*vmas)), GFP_KERNEL | __GFP_NOWARN); + if (!vmas) { + etdev_dbg(etdev, "%s: kvmalloc vmas failed (%lu bytes)\n", + __func__, (num_pages * sizeof(*pages))); + kvfree(pages); + return ERR_PTR(-ENOMEM); + } + ret = pin_user_pages(host_addr & PAGE_MASK, num_pages, foll_flags, + pages, vmas); + kvfree(vmas); if (ret < 0) { - etdev_dbg(etdev, "get user pages failed %u:%pK-%u: %d", + etdev_dbg(etdev, "pin_user_pages failed %u:%pK-%u: %d", group->workload_id, (void *)host_addr, num_pages, ret); num_pages = 0; @@ -1187,7 +1218,7 @@ static struct page **edgetpu_pin_user_pages(struct edgetpu_device_group *group, } if (ret < num_pages) { etdev_dbg(etdev, - "get user pages partial %u:%pK npages=%u pinned=%d", + "pin_user_pages partial %u:%pK npages=%u pinned=%d", group->workload_id, (void *)host_addr, num_pages, ret); num_pages = ret; @@ -1196,7 +1227,6 @@ static struct page **edgetpu_pin_user_pages(struct edgetpu_device_group *group, } *pnum_pages = num_pages; - return pages; error: @@ -1402,6 +1432,7 @@ int edgetpu_device_group_map(struct edgetpu_device_group *group, const u32 mmu_flags = map_to_mmu_flags(flags) | EDGETPU_MMU_HOST; int i; bool readonly; + tpu_addr_t tpu_addr; if (!valid_dma_direction(flags & EDGETPU_MAP_DIR_MASK)) return -EINVAL; @@ -1445,7 +1476,7 @@ int edgetpu_device_group_map(struct edgetpu_device_group *group, ret = edgetpu_device_group_map_iova_sgt(group, hmap); if (ret) { etdev_dbg(etdev, - "group add translation failed %u:0x%llx", + "group add translation failed %u:%#llx", group->workload_id, map->device_address); goto error; } @@ -1457,15 +1488,20 @@ int edgetpu_device_group_map(struct edgetpu_device_group *group, goto error; } + map->map_size = arg->size; + /* + * @map can be freed (by another thread) once it's added to the mappings, record the address + * before that. + */ + tpu_addr = map->device_address; ret = edgetpu_mapping_add(&group->host_mappings, map); if (ret) { - etdev_dbg(etdev, "duplicate mapping %u:0x%llx", - group->workload_id, map->device_address); + etdev_dbg(etdev, "duplicate mapping %u:%#llx", group->workload_id, tpu_addr); goto error; } mutex_unlock(&group->lock); - arg->device_address = map->device_address; + arg->device_address = tpu_addr; kvfree(pages); return 0; @@ -1507,7 +1543,7 @@ int edgetpu_device_group_unmap(struct edgetpu_device_group *group, if (!map) { edgetpu_mapping_unlock(&group->host_mappings); etdev_dbg(group->etdev, - "%s: mapping not found for workload %u: 0x%llx", + "%s: mapping not found for workload %u: %#llx", __func__, group->workload_id, tpu_addr); ret = -EINVAL; goto unlock_group; @@ -1577,7 +1613,10 @@ void edgetpu_mappings_clear_group(struct edgetpu_device_group *group) void edgetpu_group_mappings_show(struct edgetpu_device_group *group, struct seq_file *s) { - seq_printf(s, "workload %u", group->workload_id); + enum edgetpu_context_id context = + edgetpu_group_context_id_locked(group); + + seq_printf(s, "group %u", group->workload_id); switch (group->status) { case EDGETPU_DEVICE_GROUP_WAITING: case EDGETPU_DEVICE_GROUP_FINALIZED: @@ -1589,26 +1628,35 @@ void edgetpu_group_mappings_show(struct edgetpu_device_group *group, seq_puts(s, ": disbanded\n"); return; } - seq_printf(s, " context %d:\n", edgetpu_group_context_id_locked(group)); + + if (context == EDGETPU_CONTEXT_INVALID) + seq_puts(s, " context (none):\n"); + else if (context & EDGETPU_CONTEXT_DOMAIN_TOKEN) + seq_printf(s, " context detached %#x:\n", + context & ~(EDGETPU_CONTEXT_DOMAIN_TOKEN)); + else + seq_printf(s, " context mbox %d:\n", context); if (group->host_mappings.count) { - seq_puts(s, "host buffer mappings:\n"); + seq_printf(s, "host buffer mappings (%zd):\n", + group->host_mappings.count); edgetpu_mappings_show(&group->host_mappings, s); } if (group->dmabuf_mappings.count) { - seq_puts(s, "dma-buf buffer mappings:\n"); + seq_printf(s, "dma-buf buffer mappings (%zd):\n", + group->dmabuf_mappings.count); edgetpu_mappings_show(&group->dmabuf_mappings, s); } if (group->vii.cmd_queue_mem.vaddr) { seq_puts(s, "VII queues:\n"); - seq_printf(s, " 0x%llx %lu cmdq 0x%llx %pad\n", + seq_printf(s, " %#llx %lu cmdq %#llx %pad\n", group->vii.cmd_queue_mem.tpu_addr, DIV_ROUND_UP(group->vii.cmd_queue_mem.size, PAGE_SIZE), group->vii.cmd_queue_mem.host_addr, &group->vii.cmd_queue_mem.dma_addr); - seq_printf(s, " 0x%llx %lu rspq 0x%llx %pad\n", + seq_printf(s, " %#llx %lu rspq %#llx %pad\n", group->vii.resp_queue_mem.tpu_addr, DIV_ROUND_UP(group->vii.resp_queue_mem.size, PAGE_SIZE), @@ -1689,7 +1737,7 @@ out: void edgetpu_group_fatal_error_notify(struct edgetpu_device_group *group, uint error_mask) { - etdev_dbg(group->etdev, "notify group %u error 0x%x", + etdev_dbg(group->etdev, "notify group %u error %#x", group->workload_id, error_mask); mutex_lock(&group->lock); /* diff --git a/drivers/edgetpu/edgetpu-device-group.h b/drivers/edgetpu/edgetpu-device-group.h index 7ec262f..5c5eaaf 100644 --- a/drivers/edgetpu/edgetpu-device-group.h +++ b/drivers/edgetpu/edgetpu-device-group.h @@ -157,6 +157,11 @@ struct edgetpu_list_group { for (l = list_entry(etdev->groups.next, typeof(*l), list), g = l->grp; \ &l->list != &etdev->groups; \ l = list_entry(l->list.next, typeof(*l), list), g = l->grp) + +/* Loop through group->clients (hold group->lock prior). */ +#define for_each_list_group_client(c, group) \ + list_for_each_entry(c, &group->clients, list) + /* * Returns if the group is waiting for members to join. * @@ -330,6 +335,9 @@ int edgetpu_device_group_sync_buffer(struct edgetpu_device_group *group, /* Clear all mappings for a device group. */ void edgetpu_mappings_clear_group(struct edgetpu_device_group *group); +/* Return total size of all mappings for the group in bytes */ +size_t edgetpu_group_mappings_total_size(struct edgetpu_device_group *group); + /* * Return context ID for group MMU mappings. * diff --git a/drivers/edgetpu/edgetpu-dmabuf.c b/drivers/edgetpu/edgetpu-dmabuf.c index b0dc9a1..f98aafe 100644 --- a/drivers/edgetpu/edgetpu-dmabuf.c +++ b/drivers/edgetpu/edgetpu-dmabuf.c @@ -338,12 +338,12 @@ static void dmabuf_map_callback_show(struct edgetpu_mapping *map, container_of(map, struct edgetpu_dmabuf_map, map); if (IS_MIRRORED(dmap->map.flags)) - seq_printf(s, " <%s> mirrored: iova=0x%llx pages=%llu %s", + seq_printf(s, " <%s> mirrored: iova=%#llx pages=%llu %s", dmap->dmabufs[0]->exp_name, map->device_address, DIV_ROUND_UP(dmap->size, PAGE_SIZE), edgetpu_dma_dir_rw_s(map->dir)); else - seq_printf(s, " <%s> die %u: iova=0x%llx pages=%llu %s", + seq_printf(s, " <%s> die %u: iova=%#llx pages=%llu %s", dmap->dmabufs[0]->exp_name, map->die_index, map->device_address, DIV_ROUND_UP(dmap->size, PAGE_SIZE), edgetpu_dma_dir_rw_s(map->dir)); @@ -437,7 +437,7 @@ static void dmabuf_bulk_map_callback_show(struct edgetpu_mapping *map, container_of(map, struct edgetpu_dmabuf_map, map); int i; - seq_printf(s, " bulk: iova=0x%llx pages=%llu %s\n", + seq_printf(s, " bulk: iova=%#llx pages=%llu %s\n", map->device_address, DIV_ROUND_UP(bmap->size, PAGE_SIZE), edgetpu_dma_dir_rw_s(map->dir)); for (i = 0; i < bmap->num_entries; i++) { @@ -670,7 +670,7 @@ int edgetpu_map_dmabuf(struct edgetpu_device_group *group, get_dma_buf(dmabuf); dmap->dmabufs[0] = dmabuf; - dmap->size = size = dmabuf->size; + dmap->map.map_size = dmap->size = size = dmabuf->size; if (IS_MIRRORED(flags)) { for (i = 0; i < group->n_clients; i++) { etdev = edgetpu_device_group_nth_etdev(group, i); diff --git a/drivers/edgetpu/edgetpu-firmware.c b/drivers/edgetpu/edgetpu-firmware.c index f2d6a0c..31f0f4c 100644 --- a/drivers/edgetpu/edgetpu-firmware.c +++ b/drivers/edgetpu/edgetpu-firmware.c @@ -341,7 +341,7 @@ int edgetpu_firmware_run_locked(struct edgetpu_firmware *et_fw, if (ret) goto out_failed; - etdev_dbg(et_fw->etdev, "run fw %s flags=0x%x", name, flags); + etdev_dbg(et_fw->etdev, "run fw %s flags=%#x", name, flags); if (chip_fw->prepare_run) { /* Note this may recursively call us to run BL1 */ ret = chip_fw->prepare_run(et_fw, &new_fw_desc.buf); @@ -745,7 +745,7 @@ void edgetpu_firmware_mappings_show(struct edgetpu_dev *etdev, return; fw_iova_target = fw_buf->dram_tpa ? fw_buf->dram_tpa : fw_buf->dma_addr; iova = edgetpu_chip_firmware_iova(etdev); - seq_printf(s, " 0x%lx %lu fw - %pad %s\n", iova, + seq_printf(s, " %#lx %lu fw - %pad %s\n", iova, DIV_ROUND_UP(fw_buf->alloc_size, PAGE_SIZE), &fw_iova_target, fw_buf->flags & FW_ONDEV ? "dev" : ""); } diff --git a/drivers/edgetpu/edgetpu-firmware.h b/drivers/edgetpu/edgetpu-firmware.h index 1cdfaa1..f24135b 100644 --- a/drivers/edgetpu/edgetpu-firmware.h +++ b/drivers/edgetpu/edgetpu-firmware.h @@ -285,4 +285,20 @@ uint32_t edgetpu_firmware_get_cl(struct edgetpu_firmware *et_fw); /* Returns the build time of the image in seconds since 1970. */ uint64_t edgetpu_firmware_get_build_time(struct edgetpu_firmware *et_fw); +/* + * Kernel verify firmware signature (if EDGETPU_FEATURE_FW_SIG enabled). + * + * @etdev: the edgetpu_dev for which the initial load of a (probably + * shared) firmware image is requested + * @name: name of the image being validated (request_firmware path) + * @image_data: passes in the pointer to the raw image with signature, returns + * pointer to the firmware code image. + * @image_size: passes in the size of the raw image with signature, returns + * size of the firmware code image. + */ +bool edgetpu_firmware_verify_signature(struct edgetpu_dev *etdev, + const char *name, + void **image_data, size_t *image_size); + + #endif /* __EDGETPU_FIRMWARE_H__ */ diff --git a/drivers/edgetpu/edgetpu-fs.c b/drivers/edgetpu/edgetpu-fs.c index 0dcad64..8b5a782 100644 --- a/drivers/edgetpu/edgetpu-fs.c +++ b/drivers/edgetpu/edgetpu-fs.c @@ -871,7 +871,7 @@ static void dump_statusregs_ranges( for (reg = ranges[i].firstreg; reg <= ranges[i].lastreg; reg += sizeof(val)) { val = edgetpu_dev_read_64(etdev, reg); - seq_printf(s, "0x%08x: 0x%016llx\n", reg, val); + seq_printf(s, "%#08x: %#016llx\n", reg, val); } } } @@ -891,15 +891,15 @@ static void dump_mboxes(struct seq_file *s, struct edgetpu_dev *etdev) for (offset = 0x0; offset <= 0x40; offset += sizeof(val)) { val = edgetpu_dev_read_32(etdev, base + offset); - seq_printf(s, "0x%08x: 0x%08x\n", base + offset, val); + seq_printf(s, "%#08x: %#08x\n", base + offset, val); } for (offset = 0x1000; offset <= 0x1014; offset += sizeof(val)) { val = edgetpu_dev_read_32(etdev, base + offset); - seq_printf(s, "0x%08x: 0x%08x\n", base + offset, val); + seq_printf(s, "%#08x: %#08x\n", base + offset, val); } for (offset = 0x1800; offset <= 0x1818; offset += sizeof(val)) { val = edgetpu_dev_read_32(etdev, base + offset); - seq_printf(s, "0x%08x: 0x%08x\n", base + offset, val); + seq_printf(s, "%#08x: %#08x\n", base + offset, val); } } } @@ -1013,26 +1013,122 @@ static ssize_t clients_show( { struct edgetpu_dev *etdev = dev_get_drvdata(dev); struct edgetpu_list_device_client *lc; + ssize_t len; ssize_t ret = 0; mutex_lock(&etdev->clients_lock); for_each_list_device_client(etdev, lc) { - ret += scnprintf(buf, PAGE_SIZE - ret, - "pid %d tgid %d wakelock %d\n", - lc->client->pid, lc->client->tgid, - NO_WAKELOCK(lc->client->wakelock) ? - 0 : lc->client->wakelock->req_count); - buf += ret; + struct edgetpu_device_group *group; + + mutex_lock(&lc->client->group_lock); + group = lc->client->group; + len = scnprintf(buf, PAGE_SIZE - ret, + "pid %d tgid %d group %d wakelock %d\n", + lc->client->pid, lc->client->tgid, + group ? group->workload_id : -1, + NO_WAKELOCK(lc->client->wakelock) ? + 0 : lc->client->wakelock->req_count); + mutex_unlock(&lc->client->group_lock); + buf += len; + ret += len; } mutex_unlock(&etdev->clients_lock); return ret; } static DEVICE_ATTR_RO(clients); +static ssize_t show_group(struct edgetpu_dev *etdev, + struct edgetpu_device_group *group, char *buf, + ssize_t buflen) +{ + enum edgetpu_context_id context = + edgetpu_group_context_id_locked(group); + struct edgetpu_list_group_client *lc; + ssize_t len; + ssize_t ret = 0; + + len = scnprintf(buf, buflen - ret, "group %u ", group->workload_id); + buf += len; + ret += len; + + switch (group->status) { + case EDGETPU_DEVICE_GROUP_WAITING: + len = scnprintf(buf, buflen - ret, "forming "); + buf += len; + ret += len; + break; + case EDGETPU_DEVICE_GROUP_FINALIZED: + break; + case EDGETPU_DEVICE_GROUP_ERRORED: + len = scnprintf(buf, buflen - ret, "error %#x ", + group->fatal_errors); + buf += len; + ret += len; + break; + case EDGETPU_DEVICE_GROUP_DISBANDED: + len = scnprintf(buf, buflen - ret, "disbanded\n"); + ret += len; + return ret; + } + + if (context == EDGETPU_CONTEXT_INVALID) + len = scnprintf(buf, buflen - ret, "context (none) "); + else if (context & EDGETPU_CONTEXT_DOMAIN_TOKEN) + len = scnprintf(buf, buflen - ret, "context detached %#x ", + context & ~(EDGETPU_CONTEXT_DOMAIN_TOKEN)); + else + len = scnprintf(buf, buflen - ret, "context mbox %d ", + context); + buf += len; + ret += len; + len = scnprintf(buf, buflen - ret, "vcid %u %s%s\n", + group->vcid, group->dev_inaccessible ? "i" : "", + group->ext_mailbox ? "x" : ""); + buf += len; + ret += len; + + for_each_list_group_client(lc, group) { + len = scnprintf(buf, buflen - ret, "client %s %d:%d\n", + lc->client->etiface->name, + lc->client->pid, lc->client->tgid); + buf += len; + ret += len; + } + + len = scnprintf(buf, buflen - ret, "mappings %zd %zdB\n", + group->host_mappings.count + + group->dmabuf_mappings.count, + edgetpu_group_mappings_total_size(group)); + buf += len; + ret += len; + return ret; +} + +static ssize_t groups_show( + struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct edgetpu_dev *etdev = dev_get_drvdata(dev); + struct edgetpu_device_group *group; + struct edgetpu_list_group *lg; + ssize_t ret = 0; + + mutex_lock(&etdev->groups_lock); + etdev_for_each_group(etdev, lg, group) { + edgetpu_device_group_get(group); + ret += show_group(etdev, group, buf + ret, PAGE_SIZE - ret); + edgetpu_device_group_put(group); + } + mutex_unlock(&etdev->groups_lock); + return ret; +} +static DEVICE_ATTR_RO(groups); + static struct attribute *edgetpu_dev_attrs[] = { &dev_attr_firmware_crash_count.attr, &dev_attr_watchdog_timeout_count.attr, &dev_attr_clients.attr, + &dev_attr_groups.attr, NULL, }; diff --git a/drivers/edgetpu/edgetpu-internal.h b/drivers/edgetpu/edgetpu-internal.h index b492eab..f223dcb 100644 --- a/drivers/edgetpu/edgetpu-internal.h +++ b/drivers/edgetpu/edgetpu-internal.h @@ -36,7 +36,7 @@ #include "edgetpu-usage-stats.h" #define get_dev_for_logging(etdev) \ - ((etdev)->etiface->etcdev ? (etdev)->etiface->etcdev : (etdev)->dev) + ((etdev)->etiface && (etdev)->etiface->etcdev ? (etdev)->etiface->etcdev : (etdev)->dev) #define etdev_err(etdev, fmt, ...) dev_err(get_dev_for_logging(etdev), fmt, ##__VA_ARGS__) #define etdev_warn(etdev, fmt, ...) \ @@ -84,6 +84,7 @@ struct edgetpu_coherent_mem { dma_addr_t dma_addr; /* DMA handle for downstream IOMMU, if any */ tpu_addr_t tpu_addr; /* DMA handle for TPU internal IOMMU, if any */ u64 host_addr; /* address mapped on host for debugging */ + u64 phys_addr; /* physical address, if available */ size_t size; #ifdef CONFIG_X86 bool is_set_uc; /* memory has been marked uncached on X86 */ @@ -146,7 +147,7 @@ struct edgetpu_list_device_client { struct edgetpu_client *client; }; -/* Macro to loop through etdev->clients (hold clients_lock prior). */ +/* loop through etdev->clients (hold clients_lock prior). */ #define for_each_list_device_client(etdev, c) \ list_for_each_entry(c, &etdev->clients, list) diff --git a/drivers/edgetpu/edgetpu-iremap-pool.c b/drivers/edgetpu/edgetpu-iremap-pool.c index 91b8fea..3858fc0 100644 --- a/drivers/edgetpu/edgetpu-iremap-pool.c +++ b/drivers/edgetpu/edgetpu-iremap-pool.c @@ -10,7 +10,6 @@ #include <linux/genalloc.h> #include <linux/kernel.h> #include <linux/mm.h> -#include <linux/mutex.h> #include <linux/slab.h> #include "edgetpu-internal.h" @@ -24,7 +23,6 @@ struct edgetpu_mempool { tpu_addr_t base_tpu_addr; phys_addr_t base_phys_addr; size_t granule; - struct mutex lock; }; int edgetpu_iremap_pool_create(struct edgetpu_dev *etdev, void *base_vaddr, @@ -44,8 +42,6 @@ int edgetpu_iremap_pool_create(struct edgetpu_dev *etdev, void *base_vaddr, if (!pool) return -ENOMEM; - mutex_init(&pool->lock); - pool->gen_pool = gen_pool_create(ilog2(granule), -1); if (!pool->gen_pool) { kfree(pool); @@ -88,21 +84,21 @@ int edgetpu_iremap_alloc(struct edgetpu_dev *etdev, size_t size, if (!etmempool) return edgetpu_alloc_coherent(etdev, size, mem, context_id); - mutex_lock(&etmempool->lock); + size = __ALIGN_KERNEL(size, etmempool->granule); addr = gen_pool_alloc(etmempool->gen_pool, size); - if (!addr) { - mutex_unlock(&etmempool->lock); + if (!addr) return -ENOMEM; - } + mem->vaddr = (void *)addr; offset = mem->vaddr - etmempool->base_vaddr; mem->dma_addr = etmempool->base_dma_addr + offset; mem->tpu_addr = etmempool->base_tpu_addr + offset; + mem->phys_addr = etmempool->base_phys_addr + offset; mem->size = size; - etdev_dbg(etdev, "%s @ %llx IOVA = %llx size = %zu", - __func__, (u64)mem->vaddr, mem->dma_addr, size); - mutex_unlock(&etmempool->lock); + etdev_dbg(etdev, "%s @ %pK IOVA = %#llx size = %zu", + __func__, mem->vaddr, mem->dma_addr, size); + return 0; } @@ -116,13 +112,12 @@ void edgetpu_iremap_free(struct edgetpu_dev *etdev, edgetpu_free_coherent(etdev, mem, context_id); return; } - mutex_lock(&etmempool->lock); + etdev_dbg(etdev, "%s @ %llx IOVA = %llx size = %zu", __func__, (u64)mem->vaddr, mem->dma_addr, mem->size); gen_pool_free(etmempool->gen_pool, (unsigned long)mem->vaddr, mem->size); mem->vaddr = NULL; - mutex_unlock(&etmempool->lock); } int edgetpu_iremap_mmap(struct edgetpu_dev *etdev, struct vm_area_struct *vma, diff --git a/drivers/edgetpu/edgetpu-kci.c b/drivers/edgetpu/edgetpu-kci.c index b54fed1..d262bba 100644 --- a/drivers/edgetpu/edgetpu-kci.c +++ b/drivers/edgetpu/edgetpu-kci.c @@ -103,7 +103,7 @@ edgetpu_reverse_kci_consume_response(struct edgetpu_dev *etdev, edgetpu_handle_job_lockup(etdev, resp->retval); break; default: - etdev_warn(etdev, "%s: Unrecognized KCI request: 0x%x\n", + etdev_warn(etdev, "%s: Unrecognized KCI request: %#x\n", __func__, resp->code); } } @@ -292,7 +292,7 @@ static struct edgetpu_kci_response_element *edgetpu_kci_fetch_responses( CIRCULAR_QUEUE_REAL_INDEX(tail) >= size)) { etdev_err_ratelimited( kci->mailbox->etdev, - "Invalid response queue tail: 0x%x\n", tail); + "Invalid response queue tail: %#x\n", tail); break; } @@ -467,7 +467,7 @@ int edgetpu_kci_init(struct edgetpu_mailbox_manager *mgr, kci->cmd_queue = kci->cmd_queue_mem.vaddr; mutex_init(&kci->cmd_queue_lock); - etdev_dbg(mgr->etdev, "%s: cmdq kva=%pK iova=0x%llx dma=%pad", __func__, + etdev_dbg(mgr->etdev, "%s: cmdq kva=%pK iova=%#llx dma=%pad", __func__, kci->cmd_queue_mem.vaddr, kci->cmd_queue_mem.tpu_addr, &kci->cmd_queue_mem.dma_addr); @@ -480,7 +480,7 @@ int edgetpu_kci_init(struct edgetpu_mailbox_manager *mgr, } kci->resp_queue = kci->resp_queue_mem.vaddr; spin_lock_init(&kci->resp_queue_lock); - etdev_dbg(mgr->etdev, "%s: rspq kva=%pK iova=0x%llx dma=%pad", __func__, + etdev_dbg(mgr->etdev, "%s: rspq kva=%pK iova=%#llx dma=%pad", __func__, kci->resp_queue_mem.vaddr, kci->resp_queue_mem.tpu_addr, &kci->resp_queue_mem.dma_addr); @@ -712,14 +712,14 @@ static int edgetpu_kci_send_cmd_with_data(struct edgetpu_kci *kci, return ret; memcpy(mem.vaddr, data, size); - etdev_dbg(etdev, "%s: map kva=%pK iova=0x%llx dma=%pad", __func__, mem.vaddr, mem.tpu_addr, + etdev_dbg(etdev, "%s: map kva=%pK iova=%#llx dma=%pad", __func__, mem.vaddr, mem.tpu_addr, &mem.dma_addr); cmd->dma.address = mem.tpu_addr; cmd->dma.size = size; ret = edgetpu_kci_send_cmd(kci, cmd); edgetpu_iremap_free(etdev, &mem, EDGETPU_CONTEXT_KCI); - etdev_dbg(etdev, "%s: unmap kva=%pK iova=0x%llx dma=%pad", __func__, mem.vaddr, + etdev_dbg(etdev, "%s: unmap kva=%pK iova=%#llx dma=%pad", __func__, mem.vaddr, mem.tpu_addr, &mem.dma_addr); return ret; } @@ -850,7 +850,7 @@ enum edgetpu_fw_flavor edgetpu_kci_fw_info(struct edgetpu_kci *kci, flavor = fw_info->fw_flavor; break; default: - etdev_dbg(etdev, "unrecognized fw flavor 0x%x\n", + etdev_dbg(etdev, "unrecognized fw flavor %#x\n", fw_info->fw_flavor); } } else { @@ -951,15 +951,15 @@ void edgetpu_kci_mappings_show(struct edgetpu_dev *etdev, struct seq_file *s) if (!kci || !kci->mailbox) return; - seq_printf(s, "kci context %u:\n", EDGETPU_CONTEXT_KCI); - seq_printf(s, " 0x%llx %lu cmdq - %pad\n", + seq_printf(s, "kci context mbox %u:\n", EDGETPU_CONTEXT_KCI); + seq_printf(s, " %#llx %lu cmdq - %pad\n", kci->cmd_queue_mem.tpu_addr, DIV_ROUND_UP( QUEUE_SIZE * edgetpu_kci_queue_element_size(MAILBOX_CMD_QUEUE), PAGE_SIZE), &kci->cmd_queue_mem.dma_addr); - seq_printf(s, " 0x%llx %lu rspq - %pad\n", + seq_printf(s, " %#llx %lu rspq - %pad\n", kci->resp_queue_mem.tpu_addr, DIV_ROUND_UP( QUEUE_SIZE * diff --git a/drivers/edgetpu/edgetpu-mailbox.c b/drivers/edgetpu/edgetpu-mailbox.c index 9bda7d6..56acd72 100644 --- a/drivers/edgetpu/edgetpu-mailbox.c +++ b/drivers/edgetpu/edgetpu-mailbox.c @@ -448,7 +448,7 @@ int edgetpu_mailbox_init_vii(struct edgetpu_vii *vii, } etdev_dbg(group->etdev, - "%s: mbox %u cmdq iova=0x%llx dma=%pad\n", + "%s: mbox %u cmdq iova=%#llx dma=%pad\n", __func__, mailbox->mailbox_id, vii->cmd_queue_mem.tpu_addr, &vii->cmd_queue_mem.dma_addr); ret = edgetpu_mailbox_alloc_queue(group->etdev, mailbox, @@ -464,7 +464,7 @@ int edgetpu_mailbox_init_vii(struct edgetpu_vii *vii, } etdev_dbg(group->etdev, - "%s: mbox %u rspq iova=0x%llx dma=%pad\n", + "%s: mbox %u rspq iova=%#llx dma=%pad\n", __func__, mailbox->mailbox_id, vii->resp_queue_mem.tpu_addr, &vii->resp_queue_mem.dma_addr); mailbox->internal.group = edgetpu_device_group_get(group); @@ -999,8 +999,7 @@ static int edgetpu_mailbox_external_alloc_enable(struct edgetpu_client *client, if (ret) { while (i--) { id = ext_mailbox->descriptors[i].mailbox->mailbox_id; - if (edgetpu_mailbox_deactivate(group->etdev, id)) - etdev_err(group->etdev, "Deactivate mailbox %d failed", id); + edgetpu_mailbox_deactivate(group->etdev, id); } /* * Deactivate only fails if f/w is unresponsive which will put group @@ -1046,8 +1045,7 @@ void edgetpu_mailbox_external_disable_free_locked(struct edgetpu_device_group *g for (i = 0; i < ext_mailbox->count; i++) { id = ext_mailbox->descriptors[i].mailbox->mailbox_id; etdev_dbg(group->etdev, "Disabling mailbox: %d\n", id); - if (edgetpu_mailbox_deactivate(group->etdev, id)) - etdev_err(group->etdev, "Deactivate mailbox %d failed", id); + edgetpu_mailbox_deactivate(group->etdev, id); } /* * Deactivate only fails if f/w is unresponsive which will put group @@ -1088,8 +1086,13 @@ out: int edgetpu_mailbox_disable_ext(struct edgetpu_client *client, int mailbox_id) { - int ret; + int ret = 0; + /* + * A successful enable_ext() increases the wakelock event which prevents wakelock being + * released, so theoretically the check fail here can only happen when enable_ext() is + * failed or not called before. + */ if (!edgetpu_wakelock_lock(client->wakelock)) { etdev_err(client->etdev, "Disabling mailbox %d needs wakelock acquired\n", mailbox_id); @@ -1103,10 +1106,7 @@ int edgetpu_mailbox_disable_ext(struct edgetpu_client *client, int mailbox_id) } etdev_dbg(client->etdev, "Disabling mailbox: %d\n", mailbox_id); - ret = edgetpu_mailbox_deactivate(client->etdev, mailbox_id); - if (ret) - etdev_err(client->etdev, "Deactivate mailbox %d failed: %d", mailbox_id, ret); - + edgetpu_mailbox_deactivate(client->etdev, mailbox_id); out: if (!ret) edgetpu_wakelock_dec_event_locked(client->wakelock, @@ -1139,7 +1139,7 @@ int edgetpu_mailbox_activate(struct edgetpu_dev *etdev, u32 mailbox_id, s16 vcid return ret; } -int edgetpu_mailbox_deactivate(struct edgetpu_dev *etdev, u32 mailbox_id) +void edgetpu_mailbox_deactivate(struct edgetpu_dev *etdev, u32 mailbox_id) { struct edgetpu_handshake *eh = &etdev->mailbox_manager->open_devices; const u32 bit = BIT(mailbox_id); @@ -1148,12 +1148,15 @@ int edgetpu_mailbox_deactivate(struct edgetpu_dev *etdev, u32 mailbox_id) mutex_lock(&eh->lock); if (bit & eh->fw_state) ret = edgetpu_kci_close_device(etdev->kci, mailbox_id); - if (!ret) { - eh->state &= ~bit; - eh->fw_state &= ~bit; - } + if (ret) + etdev_err(etdev, "Deactivate mailbox %d failed: %d", mailbox_id, ret); + /* + * Always clears the states, FW should never reject CLOSE_DEVICE requests unless it's + * unresponsive. + */ + eh->state &= ~bit; + eh->fw_state &= ~bit; mutex_unlock(&eh->lock); - return ret; } void edgetpu_handshake_clear_fw_state(struct edgetpu_handshake *eh) diff --git a/drivers/edgetpu/edgetpu-mailbox.h b/drivers/edgetpu/edgetpu-mailbox.h index 3fd961f..d76534a 100644 --- a/drivers/edgetpu/edgetpu-mailbox.h +++ b/drivers/edgetpu/edgetpu-mailbox.h @@ -362,7 +362,7 @@ int edgetpu_mailbox_activate(struct edgetpu_dev *etdev, u32 mailbox_id, s16 vcid /* * Similar to edgetpu_mailbox_activate() but sends CLOSE_DEVICE KCI instead. */ -int edgetpu_mailbox_deactivate(struct edgetpu_dev *etdev, u32 mailbox_id); +void edgetpu_mailbox_deactivate(struct edgetpu_dev *etdev, u32 mailbox_id); /* Sets @eh->fw_state to 0. */ void edgetpu_handshake_clear_fw_state(struct edgetpu_handshake *eh); /* diff --git a/drivers/edgetpu/edgetpu-mapping.c b/drivers/edgetpu/edgetpu-mapping.c index 136cf29..fbffcce 100644 --- a/drivers/edgetpu/edgetpu-mapping.c +++ b/drivers/edgetpu/edgetpu-mapping.c @@ -152,3 +152,21 @@ void edgetpu_mappings_show(struct edgetpu_mapping_root *mappings, edgetpu_mapping_unlock(mappings); } + +size_t edgetpu_mappings_total_size(struct edgetpu_mapping_root *mappings) +{ + struct rb_node *node; + size_t total = 0; + + edgetpu_mapping_lock(mappings); + + for (node = rb_first(&mappings->rb); node; node = rb_next(node)) { + struct edgetpu_mapping *map = + container_of(node, struct edgetpu_mapping, node); + + total += map->map_size; + } + + edgetpu_mapping_unlock(mappings); + return total; +} diff --git a/drivers/edgetpu/edgetpu-mapping.h b/drivers/edgetpu/edgetpu-mapping.h index 8ad165b..8f075aa 100644 --- a/drivers/edgetpu/edgetpu-mapping.h +++ b/drivers/edgetpu/edgetpu-mapping.h @@ -40,6 +40,8 @@ struct edgetpu_mapping { u64 host_address; u32 die_index; /* this mapping is mapped on the @die_index-th die */ tpu_addr_t device_address; + /* Size of buffer mapped in bytes. Always set. */ + size_t map_size; /* * The size used for allocating @alloc_iova in bytes. This field may be * set by edgetpu_mmu_map(). @@ -174,4 +176,7 @@ static inline int mmu_flag_to_iommu_prot(u32 mmu_flags, struct device *dev, return prot; } +/* Return total size of mappings under the supplied root. */ +size_t edgetpu_mappings_total_size(struct edgetpu_mapping_root *mappings); + #endif /* __EDGETPU_MAPPING_H__ */ diff --git a/drivers/edgetpu/edgetpu-mobile-firmware.c b/drivers/edgetpu/edgetpu-mobile-firmware.c index 407a698..dedac0f 100644 --- a/drivers/edgetpu/edgetpu-mobile-firmware.c +++ b/drivers/edgetpu/edgetpu-mobile-firmware.c @@ -37,7 +37,7 @@ int edgetpu_firmware_chip_load_locked( aligned_size = ALIGN(fw->size, fw_desc->buf.used_size_align); if (aligned_size > fw_desc->buf.alloc_size) { etdev_dbg(etdev, - "%s: firmware buffer too small: alloc size=0x%zx, required size=0x%zx\n", + "%s: firmware buffer too small: alloc size=%#zx, required size=%#zx\n", __func__, fw_desc->buf.alloc_size, aligned_size); ret = -ENOSPC; goto out_release_firmware; diff --git a/drivers/edgetpu/edgetpu-shared-fw.c b/drivers/edgetpu/edgetpu-shared-fw.c deleted file mode 100644 index c41215e..0000000 --- a/drivers/edgetpu/edgetpu-shared-fw.c +++ /dev/null @@ -1,284 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Edge TPU shared firmware management. - * - * Copyright (C) 2020 Google, Inc. - */ - -#include <linux/device.h> -#include <linux/firmware.h> -#include <linux/list.h> -#include <linux/mutex.h> -#include <linux/refcount.h> -#include <linux/slab.h> -#include <linux/string.h> - -#include "edgetpu-firmware.h" -#include "edgetpu-firmware-util.h" -#include "edgetpu-internal.h" -#include "edgetpu-shared-fw.h" - -struct edgetpu_shared_fw_buffer { - /* - * Shared firmware buffer is managed by `global.firmware_list`, so that - * each data member is protected by `global.lock`. - */ - struct list_head list; - /* - * Counting for devices holding the buffer. We can only release the data - * buffer if there is no device nor sysfs holding the firmware. - * - * Even when the reference count atomically decreased down to 0, there's - * a chance that someone is traversing list and trying to read this - * `ref`. So `ref` must still be protected by `glock.lock` in this - * case. - */ - refcount_t ref; - /* - * Indicates if this buffer is loaded by sysfs. - * - * Reference count caused by sysfs load should be exactly 1, and we can - * only unload firmware by sysfs if already loaded by sysfs. - */ - bool is_sysfs_loaded; - /* Firmware name, the same as that loaded by request_firmware() API. */ - const char *name; - void *vaddr; - /* The size of buffer is aligned to `global.init_data.size_align`. */ - size_t size; -}; - -const char * -edgetpu_shared_fw_buffer_name(const struct edgetpu_shared_fw_buffer *buffer) -{ - return buffer->name; -} - -void * -edgetpu_shared_fw_buffer_vaddr(const struct edgetpu_shared_fw_buffer *buffer) -{ - return buffer->vaddr; -} - -size_t -edgetpu_shared_fw_buffer_size(const struct edgetpu_shared_fw_buffer *buffer) -{ - return buffer->size; -} - -/* - * Lock protected global data. - * - * global.lock is required for invoking _locked functions in this file. - */ -static struct { - struct mutex lock; - struct edgetpu_shared_fw_init_data init_data; - struct list_head firmware_list; -} global = { - .lock = __MUTEX_INITIALIZER(global.lock), - .firmware_list = LIST_HEAD_INIT(global.firmware_list), -}; - -#define for_each_shared_fw_buffer(buffer) \ - list_for_each_entry(buffer, &global.firmware_list, list) -#define for_each_shared_fw_buffer_safe(cur_buf, nxt_buf) \ - list_for_each_entry_safe(cur_buf, nxt_buf, &global.firmware_list, list) - -static struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_find_locked(const char *name) -{ - struct edgetpu_shared_fw_buffer *buffer; - - for_each_shared_fw_buffer(buffer) { - if (!strcmp(name, buffer->name)) - return buffer; - } - return NULL; -} - -void -edgetpu_shared_fw_init(const struct edgetpu_shared_fw_init_data *init_data) -{ - if (list_empty(&global.firmware_list)) - global.init_data = *init_data; -} - -void edgetpu_shared_fw_exit(void) -{ - struct edgetpu_shared_fw_buffer *cur_buf, *nxt_buf; - - mutex_lock(&global.lock); - - if (!list_empty(&global.firmware_list)) - pr_warn("%s: firmware not released on exiting\n", __func__); - - for_each_shared_fw_buffer_safe(cur_buf, nxt_buf) - list_del(&cur_buf->list); - - mutex_unlock(&global.lock); -} - -static struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_get_locked(struct edgetpu_shared_fw_buffer *buffer) -{ - if (!buffer) - return NULL; - if (!refcount_inc_not_zero(&buffer->ref)) - return NULL; - return buffer; -} - -struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_get(struct edgetpu_shared_fw_buffer *buffer) -{ - mutex_lock(&global.lock); - buffer = edgetpu_shared_fw_get_locked(buffer); - mutex_unlock(&global.lock); - return buffer; -} - -struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_get_by_name(const char *name) -{ - struct edgetpu_shared_fw_buffer *buffer; - - mutex_lock(&global.lock); - buffer = edgetpu_shared_fw_get_locked( - edgetpu_shared_fw_find_locked(name)); - mutex_unlock(&global.lock); - return buffer; -} - -static struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_load_locked(const char *name, struct edgetpu_dev *etdev) -{ - int ret; - const struct firmware *fw; - size_t aligned_size; - struct edgetpu_shared_fw_buffer *buffer; - - buffer = edgetpu_shared_fw_get_locked( - edgetpu_shared_fw_find_locked(name)); - if (buffer) { - pr_debug("%s: found shared fw image %s\n", __func__, name); - return buffer; - } - - pr_debug("%s: shared fw image %s not found, requesting\n", - __func__, name); - ret = request_firmware(&fw, name, etdev ? etdev->etiface->etcdev : NULL); - if (ret) - goto out; - - aligned_size = ALIGN(fw->size, global.init_data.size_align); - - buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); - if (!buffer) { - ret = -ENOMEM; - goto out_release_firmware; - } - - buffer->name = kstrdup(name, GFP_KERNEL); - if (!buffer->name) { - ret = -ENOMEM; - goto out_kfree_buffer; - } - - /* Allocated in page alignment for mmu and dma mapping. */ - if (aligned_size < PAGE_SIZE) - buffer->vaddr = kmalloc_order(aligned_size, GFP_KERNEL, 1); - else - buffer->vaddr = kmalloc(aligned_size, GFP_KERNEL); - if (!buffer->vaddr) { - ret = -ENOMEM; - goto out_kfree_buffer_name; - } - memcpy(buffer->vaddr, fw->data, fw->size); - release_firmware(fw); - - buffer->size = aligned_size; - refcount_set(&buffer->ref, 1); - - list_add(&buffer->list, &global.firmware_list); - return buffer; - -out_kfree_buffer_name: - kfree(buffer->name); -out_kfree_buffer: - kfree(buffer); -out_release_firmware: - release_firmware(fw); -out: - return ERR_PTR(ret); -} - -struct edgetpu_shared_fw_buffer *edgetpu_shared_fw_load( - const char *name, struct edgetpu_dev *etdev) -{ - struct edgetpu_shared_fw_buffer *buffer; - - mutex_lock(&global.lock); - buffer = edgetpu_shared_fw_load_locked(name, etdev); - mutex_unlock(&global.lock); - return buffer; -} - -int edgetpu_firmware_chip_load_locked( - struct edgetpu_firmware *et_fw, - struct edgetpu_firmware_desc *fw_desc, const char *name) -{ - int ret; - struct edgetpu_dev *etdev = et_fw->etdev; - struct edgetpu_shared_fw_buffer *shared_buf; - - shared_buf = edgetpu_shared_fw_load(name, etdev); - if (IS_ERR(shared_buf)) { - ret = PTR_ERR(shared_buf); - etdev_dbg(etdev, "shared buffer loading failed: %d\n", ret); - return ret; - } - fw_desc->shared_buf = shared_buf; - fw_desc->buf.vaddr = edgetpu_shared_fw_buffer_vaddr(shared_buf); - fw_desc->buf.alloc_size = edgetpu_shared_fw_buffer_size(shared_buf); - fw_desc->buf.used_size = fw_desc->buf.alloc_size; - fw_desc->buf.name = edgetpu_shared_fw_buffer_name(shared_buf); - return 0; -} - -void edgetpu_firmware_chip_unload_locked( - struct edgetpu_firmware *et_fw, - struct edgetpu_firmware_desc *fw_desc) -{ - fw_desc->buf.vaddr = NULL; - fw_desc->buf.alloc_size = 0; - fw_desc->buf.used_size = 0; - fw_desc->buf.name = NULL; - edgetpu_shared_fw_put(fw_desc->shared_buf); - fw_desc->shared_buf = NULL; -} - -static void -edgetpu_shared_fw_put_locked(struct edgetpu_shared_fw_buffer *buffer) -{ - if (!buffer) - return; - - /* - * buffer->ref IS protected by global.lock. See also `ref` in `struct - * edgetpu_shared_fw_buffer`. - */ - if (refcount_dec_and_test(&buffer->ref)) { - kfree(buffer->vaddr); - kfree(buffer->name); - list_del(&buffer->list); - kfree(buffer); - } -} - -void edgetpu_shared_fw_put(struct edgetpu_shared_fw_buffer *buffer) -{ - mutex_lock(&global.lock); - edgetpu_shared_fw_put_locked(buffer); - mutex_unlock(&global.lock); -} diff --git a/drivers/edgetpu/edgetpu-shared-fw.h b/drivers/edgetpu/edgetpu-shared-fw.h deleted file mode 100644 index 28033e3..0000000 --- a/drivers/edgetpu/edgetpu-shared-fw.h +++ /dev/null @@ -1,74 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * Implements utilities for shared firmware management of EdgeTPU. - * - * Copyright (C) 2020 Google, Inc. - */ -#ifndef __EDGETPU_SHARED_FW_H__ -#define __EDGETPU_SHARED_FW_H__ - -#include <linux/device.h> - -#include "edgetpu-internal.h" - -struct edgetpu_shared_fw_buffer; - -/* - * name for this firmware in null terminated string, the same as which loaded by - * linux request_firmware API - */ -const char * -edgetpu_shared_fw_buffer_name(const struct edgetpu_shared_fw_buffer *buffer); -/* host address for this firmware */ -void * -edgetpu_shared_fw_buffer_vaddr(const struct edgetpu_shared_fw_buffer *buffer); -/* size in bytes for this firmware */ -size_t -edgetpu_shared_fw_buffer_size(const struct edgetpu_shared_fw_buffer *buffer); - -struct edgetpu_shared_fw_init_data { - /* firmware size alignment in bytes */ - size_t size_align; -}; - -/* Initializes structures for shared firmware management. */ -void -edgetpu_shared_fw_init(const struct edgetpu_shared_fw_init_data *init_data); -/* Finalizes structures for shared firmware management. */ -void edgetpu_shared_fw_exit(void); - -/* - * Load reference counted shared firmware from file system. Increase reference - * count by 1 if the firmware is already loaded before. - * - * Firmware loaded by this function should be released by - * edgetpu_shared_fw_put(). - * - * @name: firmware path to be loaded - * @etdev: requesting edgetpu_dev, if any, for logging - */ -struct edgetpu_shared_fw_buffer *edgetpu_shared_fw_load( - const char *name, struct edgetpu_dev *etdev); - -/* - * Increase the reference count of the buffer by 1. - * - * returns the buffer, behave the same as other *_get/put() functions - */ -struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_get(struct edgetpu_shared_fw_buffer *buffer); -/* - * Find the shared firmware by name and increase the reference count of the - * found buffer by 1. - * - * returns NULL on error or not found - */ -struct edgetpu_shared_fw_buffer * -edgetpu_shared_fw_get_by_name(const char *name); - -/* - * Decrease the reference count by 1 and free the shared buffer if its - * reference count reaches 0. - */ -void edgetpu_shared_fw_put(struct edgetpu_shared_fw_buffer *buffer); -#endif /* __EDGETPU_SHARED_FW_H__ */ diff --git a/drivers/edgetpu/edgetpu-telemetry.c b/drivers/edgetpu/edgetpu-telemetry.c index 4e1053e..01c34a9 100644 --- a/drivers/edgetpu/edgetpu-telemetry.c +++ b/drivers/edgetpu/edgetpu-telemetry.c @@ -243,7 +243,7 @@ static void telemetry_mappings_show(struct edgetpu_telemetry *tel, if (!tel->inited) return; - seq_printf(s, " 0x%llx %lu %s 0x%llx %pad\n", + seq_printf(s, " %#llx %lu %s %#llx %pad\n", tel->coherent_mem.tpu_addr, DIV_ROUND_UP(tel->coherent_mem.size, PAGE_SIZE), tel->name, tel->coherent_mem.host_addr, &tel->coherent_mem.dma_addr); diff --git a/drivers/edgetpu/edgetpu-thermal.h b/drivers/edgetpu/edgetpu-thermal.h index 8d4b23f..dbd283f 100644 --- a/drivers/edgetpu/edgetpu-thermal.h +++ b/drivers/edgetpu/edgetpu-thermal.h @@ -23,6 +23,7 @@ struct edgetpu_thermal { struct mutex lock; void *op_data; unsigned long cooling_state; + unsigned long sysfs_req; unsigned int tpu_num_states; struct edgetpu_dev *etdev; bool thermal_suspended; /* TPU thermal suspended state */ diff --git a/drivers/edgetpu/edgetpu.h b/drivers/edgetpu/edgetpu.h index 3f43bab..5a630a4 100644 --- a/drivers/edgetpu/edgetpu.h +++ b/drivers/edgetpu/edgetpu.h @@ -296,7 +296,7 @@ struct edgetpu_map_dmabuf_ioctl { */ __u64 device_address; /* A dma-buf FD. */ - int dmabuf_fd; + __s32 dmabuf_fd; /* * Flags indicating mapping attributes. See edgetpu_map_ioctl.flags for * details. diff --git a/drivers/edgetpu/include/linux/platform_data/sscoredump.h b/drivers/edgetpu/include/linux/platform_data/sscoredump.h deleted file mode 100644 index fee2872..0000000 --- a/drivers/edgetpu/include/linux/platform_data/sscoredump.h +++ /dev/null @@ -1,35 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -#ifndef __SUBSYSTEM_COREDUMP_H -#define __SUBSYSTEM_COREDUMP_H - -#include <linux/device.h> -#include <linux/module.h> -#include <linux/vmalloc.h> -#include <linux/platform_device.h> - -#define SSCD_NAME "sscoredump" - -/* sscd segment ( similar to ELF memory segments) */ -struct sscd_segment { - void *addr; - u64 size; - u64 flags; - - /* passed to elf sprogram header */ - void *paddr; - void *vaddr; -} __packed; - -/* sscd_report flags */ -#define SSCD_FLAGS_ELFARM32HDR 0x0001 -#define SSCD_FLAGS_ELFARM64HDR 0x0002 - - -struct sscd_platform_data { - /* report crash */ - int (*sscd_report)(struct platform_device *pdev, - struct sscd_segment *segs, int nsegs, - u64 flags, const char *crash_info); -}; - -#endif /* __SUBSYSTEM_COREDUMP_H */ diff --git a/drivers/edgetpu/janeiro-debug-dump.c b/drivers/edgetpu/janeiro-debug-dump.c index d38d7e6..394ec08 100644 --- a/drivers/edgetpu/janeiro-debug-dump.c +++ b/drivers/edgetpu/janeiro-debug-dump.c @@ -1,5 +1,11 @@ // SPDX-License-Identifier: GPL-2.0 -#include "edgetpu-debug-dump.c" +/* + * Implements chip specific details of debug dump memory initialization and SSCD registration. + * + * Copyright (C) 2021 Google, Inc. + */ + +#include "mobile-debug-dump.c" int edgetpu_debug_dump_init(struct edgetpu_dev *etdev) { diff --git a/drivers/edgetpu/janeiro-device.c b/drivers/edgetpu/janeiro-device.c index 7b1aa6d..36dbe76 100644 --- a/drivers/edgetpu/janeiro-device.c +++ b/drivers/edgetpu/janeiro-device.c @@ -15,6 +15,14 @@ #include "janeiro-platform.h" #include "mobile-pm.h" +#define SSMT_NS_READ_STREAM_VID_OFFSET(n) (0x1000u + (0x4u * (n))) +#define SSMT_NS_WRITE_STREAM_VID_OFFSET(n) (0x1200u + (0x4u * (n))) + +#define SSMT_NS_READ_STREAM_VID_REG(base, n) \ + ((base) + SSMT_NS_READ_STREAM_VID_OFFSET(n)) +#define SSMT_NS_WRITE_STREAM_VID_REG(base, n) \ + ((base) + SSMT_NS_WRITE_STREAM_VID_OFFSET(n)) + static irqreturn_t janeiro_mailbox_handle_irq(struct edgetpu_dev *etdev, int irq) { @@ -51,10 +59,6 @@ irqreturn_t edgetpu_chip_irq_handler(int irq, void *arg) struct edgetpu_dev *etdev = arg; edgetpu_telemetry_irq_handler(etdev); - /* - * use this as HOST_NONSECURE_INT_SRC_STATUS_REG not present in - * Janeiro. - */ return janeiro_mailbox_handle_irq(etdev, irq); } @@ -65,6 +69,17 @@ u64 edgetpu_chip_tpu_timestamp(struct edgetpu_dev *etdev) void edgetpu_chip_init(struct edgetpu_dev *etdev) { + int i; + struct janeiro_platform_dev *jpdev = to_janeiro_dev(etdev); + + if (!jpdev->ssmt_base) + return; + + /* Setup non-secure SCIDs, assume VID = SCID */ + for (i = 0; i < EDGETPU_NCONTEXTS; i++) { + writel(i, SSMT_NS_READ_STREAM_VID_REG(jpdev->ssmt_base, i)); + writel(i, SSMT_NS_WRITE_STREAM_VID_REG(jpdev->ssmt_base, i)); + } } void edgetpu_chip_exit(struct edgetpu_dev *etdev) diff --git a/drivers/edgetpu/janeiro-firmware.c b/drivers/edgetpu/janeiro-firmware.c index 3ce8880..ef43931 100644 --- a/drivers/edgetpu/janeiro-firmware.c +++ b/drivers/edgetpu/janeiro-firmware.c @@ -133,7 +133,7 @@ static int janeiro_firmware_setup_buffer(struct edgetpu_firmware *et_fw, { int ret = 0; void *image_vaddr; - u32 tpu_addr, phys_addr, size, i; + u32 tpu_addr, phys_addr, size, i, j; struct janeiro_image_config *image_config; struct janeiro_firmware_data *data; struct edgetpu_dev *etdev = et_fw->etdev; @@ -168,24 +168,32 @@ static int janeiro_firmware_setup_buffer(struct edgetpu_firmware *et_fw, edgetpu_mmu_remove_translation(etdev, tpu_addr, size, EDGETPU_CONTEXT_KCI); } - for (i = 0; i < image_config->num_iommu_mapping; i++) { + for (i = 0, j = 0; i < MAX_IOMMU_MAPPINGS; i++) { tpu_addr = image_config->mappings[i].virt_address; + if (!tpu_addr) + continue; size = CONFIG_TO_SIZE(image_config->mappings[i].image_config_value); phys_addr = (image_config->mappings[i].image_config_value & ~(0xFFF)); ret = edgetpu_mmu_add_translation(etdev, tpu_addr, phys_addr, size, IOMMU_READ | IOMMU_WRITE, EDGETPU_CONTEXT_KCI); if (ret) { - etdev_err(etdev, - "Unable to Map: %d tpu_addr: 0x%x phys_addr: 0x%x size: 0x%x\n", - ret, tpu_addr, phys_addr, size); + etdev_dbg(etdev, + "Unable to map: %d tpu_addr: %#x phys_addr: %pap size: %#x\n", + ret, tpu_addr, &phys_addr, size); goto err; } - data->mappings[i].virt_address = tpu_addr; - data->mappings[i].image_config_value = image_config->mappings[i].image_config_value; + data->mappings[j].virt_address = tpu_addr; + data->mappings[j++].image_config_value = + image_config->mappings[i].image_config_value; } - data->num_mapping = image_config->num_iommu_mapping; + if (image_config->num_iommu_mapping != j) { + etdev_err(etdev, "Invalid firmware header\n"); + ret = -EINVAL; + goto err; + } + data->num_mapping = j; /* Skip the header */ memcpy(image_vaddr, fw_buf->vaddr + MOBILE_FW_HEADER_SIZE, @@ -193,9 +201,9 @@ static int janeiro_firmware_setup_buffer(struct edgetpu_firmware *et_fw, memunmap(image_vaddr); return 0; err: - while (i--) { - tpu_addr = data->mappings[i].virt_address; - size = CONFIG_TO_SIZE(data->mappings[i].image_config_value); + while (j--) { + tpu_addr = data->mappings[j].virt_address; + size = CONFIG_TO_SIZE(data->mappings[j].image_config_value); edgetpu_mmu_remove_translation(etdev, tpu_addr, size, EDGETPU_CONTEXT_KCI); } data->num_mapping = 0; diff --git a/drivers/edgetpu/janeiro-platform.c b/drivers/edgetpu/janeiro-platform.c index 73c6119..5708a5d 100644 --- a/drivers/edgetpu/janeiro-platform.c +++ b/drivers/edgetpu/janeiro-platform.c @@ -117,6 +117,32 @@ void edgetpu_chip_remove_mmu(struct edgetpu_dev *etdev) edgetpu_mmu_detach(etdev); } +static int janeiro_parse_ssmt(struct janeiro_platform_dev *etpdev) +{ + struct edgetpu_dev *etdev = &etpdev->edgetpu_dev; + struct platform_device *pdev = to_platform_device(etdev->dev); + struct resource *res; + int rc; + void __iomem *ssmt_base; + + /* TODO(b/197301774): Remove when GSA configure SSMT */ + return -EINVAL; + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "ssmt"); + if (!res) { + etdev_warn(etdev, "Failed to find SSMT register base"); + return -EINVAL; + } + ssmt_base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(ssmt_base)) { + rc = PTR_ERR(ssmt_base); + etdev_warn(etdev, "Failed to map SSMT register base: %d\n", rc); + return rc; + } + etpdev->ssmt_base = ssmt_base; + return 0; +} + /* * Set shareability for enabling IO coherency in Janeiro */ @@ -263,6 +289,13 @@ static int edgetpu_platform_probe(struct platform_device *pdev) goto out_remove_device; } + ret = janeiro_parse_ssmt(edgetpu_pdev); + if (ret) + dev_warn( + dev, + "SSMT setup failed (%d). Context isolation not enforced\n", + ret); + janeiro_get_telemetry_mem(edgetpu_pdev, EDGETPU_TELEMETRY_LOG, &edgetpu_pdev->log_mem); janeiro_get_telemetry_mem(edgetpu_pdev, EDGETPU_TELEMETRY_TRACE, diff --git a/drivers/edgetpu/janeiro-platform.h b/drivers/edgetpu/janeiro-platform.h index 09991cf..51eb8d6 100644 --- a/drivers/edgetpu/janeiro-platform.h +++ b/drivers/edgetpu/janeiro-platform.h @@ -11,7 +11,10 @@ #include <linux/io.h> #include <linux/mutex.h> #include <linux/types.h> + +#if IS_ENABLED(CONFIG_GOOGLE_BCL) #include <soc/google/bcl.h> +#endif #include "edgetpu-internal.h" diff --git a/drivers/edgetpu/janeiro-pm.c b/drivers/edgetpu/janeiro-pm.c index e69f774..700601e 100644 --- a/drivers/edgetpu/janeiro-pm.c +++ b/drivers/edgetpu/janeiro-pm.c @@ -10,9 +10,12 @@ #include <linux/iopoll.h> #include <linux/module.h> #include <linux/pm_runtime.h> -#include <soc/google/bcl.h> #include <linux/version.h> +#if IS_ENABLED(CONFIG_GOOGLE_BCL) +#include <soc/google/bcl.h> +#endif + #include "edgetpu-firmware.h" #include "edgetpu-internal.h" #include "edgetpu-kci.h" @@ -25,9 +28,9 @@ #define SHUTDOWN_DELAY_US_MIN 20 #define SHUTDOWN_DELAY_US_MAX 20 -#define BOOTUP_DELAY_US_MIN 100 -#define BOOTUP_DELAY_US_MAX 150 -#define SHUTDOWN_MAX_DELAY_COUNT 20 +#define BOOTUP_DELAY_US_MIN 200 +#define BOOTUP_DELAY_US_MAX 250 +#define SHUTDOWN_MAX_DELAY_COUNT 50 /* Default power state */ static int power_state = TPU_ACTIVE_NOM; @@ -112,7 +115,8 @@ static int janeiro_pwr_state_set_locked(void *data, u64 val) do { /* Delay 20us per retry till blk shutdown finished */ usleep_range(SHUTDOWN_DELAY_US_MIN, SHUTDOWN_DELAY_US_MAX); - curr_state = exynos_acpm_get_rate(TPU_ACPM_DOMAIN, 0); + /* Only poll for BLK status instead of CLK rate */ + curr_state = exynos_acpm_get_rate(TPU_ACPM_DOMAIN, 1); if (!curr_state) break; timeout_cnt++; @@ -237,14 +241,14 @@ static int janeiro_set_lpm(struct edgetpu_dev *etdev) edgetpu_dev_write_32_sync(etdev, EDGETPU_PSM0_START, 1); ret = readl_poll_timeout(etdev->regs.mem + EDGETPU_PSM0_STATUS, val, - val & 0x80, 5, EDGETPU_LPM_CHANGE_TIMEOUT); + val & 0x80, 20, EDGETPU_LPM_CHANGE_TIMEOUT); if (ret) { etdev_err(etdev, "Set LPM0 failed: %d\n", ret); return ret; } edgetpu_dev_write_32_sync(etdev, EDGETPU_PSM1_START, 1); ret = readl_poll_timeout(etdev->regs.mem + EDGETPU_PSM1_STATUS, val, - val & 0x80, 5, EDGETPU_LPM_CHANGE_TIMEOUT); + val & 0x80, 20, EDGETPU_LPM_CHANGE_TIMEOUT); if (ret) { etdev_err(etdev, "Set LPM1 failed: %d\n", ret); return ret; diff --git a/drivers/edgetpu/janeiro-pm.h b/drivers/edgetpu/janeiro-pm.h deleted file mode 100644 index 78991cc..0000000 --- a/drivers/edgetpu/janeiro-pm.h +++ /dev/null @@ -1,79 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * Power management header for Janeiro. - * - * Copyright (C) 2020 Google, Inc. - */ -#ifndef __JANEIRO_PM_H__ -#define __JANEIRO_PM_H__ - -#include "edgetpu-internal.h" -#include "edgetpu-kci.h" - -/* Can't build out of tree with acpm_dvfs unless kernel supports ACPM */ -#if IS_ENABLED(CONFIG_ACPM_DVFS) - -#include <linux/acpm_dvfs.h> - -#else - -static unsigned long exynos_acpm_rate; -static inline int exynos_acpm_set_rate(unsigned int id, unsigned long rate) -{ - exynos_acpm_rate = rate; - return 0; -} -static inline int exynos_acpm_set_init_freq(unsigned int dfs_id, - unsigned long freq) -{ - return 0; -} -static inline unsigned long exynos_acpm_get_rate(unsigned int id, - unsigned long dbg_val) -{ - return exynos_acpm_rate; -} -static inline int exynos_acpm_set_policy(unsigned int id, unsigned long policy) -{ - return 0; -} -#endif /* IS_ENABLED(CONFIG_ACPM_DVFS) */ -//TODO(b/185797093): check abrolhos ported values for janeiro -/* - * TPU Power States: - * 0: Off - * 227000 Ultra Underdrive @227MHz - * 625000: Super Underdrive @625MHz - * 845000: Underdrive @845MHz - * 1066000: Nominal @1066MHz - */ -enum tpu_pwr_state { - TPU_OFF = 0, - TPU_ACTIVE_UUD = 227000, - TPU_ACTIVE_SUD = 625000, - TPU_ACTIVE_UD = 845000, - TPU_ACTIVE_NOM = 1066000, -}; - -/* - * Request codes from firmware - * Values must match with firmware code base - */ -enum janeiro_reverse_kci_code { - RKCI_CODE_PM_QOS = RKCI_CHIP_CODE_FIRST + 1, - RKCI_CODE_BTS = RKCI_CHIP_CODE_FIRST + 2, -}; - -#define TPU_POLICY_MAX TPU_ACTIVE_NOM - -#define TPU_ACPM_DOMAIN 7 - -int janeiro_pm_create(struct edgetpu_dev *etdev); - -void janeiro_pm_destroy(struct edgetpu_dev *etdev); - -void janeiro_pm_set_pm_qos(struct edgetpu_dev *etdev, u32 pm_qos_val); - -void janeiro_pm_set_bts(struct edgetpu_dev *etdev, u32 bts_val); - -#endif /* __JANEIRO_PM_H__ */ diff --git a/drivers/edgetpu/janeiro/config-mailbox.h b/drivers/edgetpu/janeiro/config-mailbox.h index e2a07cf..6d7bce9 100644 --- a/drivers/edgetpu/janeiro/config-mailbox.h +++ b/drivers/edgetpu/janeiro/config-mailbox.h @@ -30,10 +30,6 @@ #define JANEIRO_CSR_MBOX_CMD_QUEUE_DOORBELL_SET_OFFSET 0x1000 #define JANEIRO_CSR_MBOX_RESP_QUEUE_DOORBELL_SET_OFFSET 0x1800 #define EDGETPU_MBOX_BASE JANEIRO_CSR_MBOX2_CONTEXT_ENABLE -// TODO: check correct values -/* CSR storing mailbox response queue doorbell status */ -#define HOST_NONSECURE_INT_SRC_STATUS_REG 0x000f0000 -#define HOST_NONSECURE_INT_SRC_CLEAR_REG 0x000f0008 static inline u32 edgetpu_mailbox_get_context_csr_base(u32 index) { diff --git a/drivers/edgetpu/mobile-debug-dump.c b/drivers/edgetpu/mobile-debug-dump.c new file mode 100644 index 0000000..82dc5a2 --- /dev/null +++ b/drivers/edgetpu/mobile-debug-dump.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Implements methods common to the family of EdgeTPUs for mobile devices to retrieve host side + * debug dump segments and report them to SSCD. + * + * Copyright (C) 2021 Google, Inc. + */ + +#include <linux/mutex.h> +#include <linux/rbtree.h> +#include <linux/slab.h> + +#include "edgetpu-device-group.h" +#include "edgetpu-mailbox.h" +#include "mobile-debug-dump.h" + +#include "edgetpu-debug-dump.c" + +struct mobile_sscd_mappings_dump * +mobile_sscd_collect_mappings_segment(struct edgetpu_device_group **groups, size_t num_groups, + struct sscd_segment *sscd_seg) +{ + struct mobile_sscd_mappings_dump *mappings_dump; + struct edgetpu_mapping_root *mappings; + struct rb_node *node; + void *resized_arr; + size_t idx = 0, mappings_num = 0, new_size = 0; + + mappings_dump = kmalloc(sizeof(struct mobile_sscd_mappings_dump), GFP_KERNEL); + for (idx = 0; idx < num_groups; idx++) { + mutex_lock(&groups[idx]->lock); + new_size += groups[idx]->host_mappings.count * + sizeof(struct mobile_sscd_mappings_dump); + resized_arr = krealloc(mappings_dump, new_size, GFP_KERNEL); + if (!resized_arr) { + kfree(mappings_dump); + mutex_unlock(&groups[idx]->lock); + return NULL; + } + + mappings = &groups[idx]->host_mappings; + for (node = rb_first(&mappings->rb); node; node = rb_next(node)) { + struct edgetpu_mapping *map = + container_of(node, struct edgetpu_mapping, node); + + mappings_dump[mappings_num].host_address = map->host_address; + mappings_dump[mappings_num].device_address = map->device_address; + mappings_dump[mappings_num].alloc_iova = map->alloc_iova; + mappings_dump[mappings_num].size = (u64)map->alloc_size; + mappings_num++; + } + new_size += groups[idx]->dmabuf_mappings.count * + sizeof(struct mobile_sscd_mappings_dump); + resized_arr = krealloc(mappings_dump, new_size, GFP_KERNEL); + if (!resized_arr) { + kfree(mappings_dump); + mutex_unlock(&groups[idx]->lock); + return NULL; + } + + mappings = &groups[idx]->dmabuf_mappings; + for (node = rb_first(&mappings->rb); node; node = rb_next(node)) { + struct edgetpu_mapping *map = + container_of(node, struct edgetpu_mapping, node); + + mappings_dump[mappings_num].host_address = map->host_address; + mappings_dump[mappings_num].device_address = map->device_address; + mappings_dump[mappings_num].alloc_iova = map->alloc_iova; + mappings_dump[mappings_num].size = (u64)map->alloc_size; + mappings_num++; + } + mutex_unlock(&groups[idx]->lock); + } + + sscd_seg->addr = mappings_dump; + sscd_seg->size = new_size; + sscd_seg->vaddr = mappings_dump; + + return mappings_dump; +} + +size_t mobile_sscd_collect_cmd_resp_queues(struct edgetpu_dev *etdev, + struct edgetpu_device_group **groups, size_t num_groups, + struct sscd_segment *sscd_seg_arr) +{ + struct edgetpu_kci *kci; + size_t idx; + u16 num_queues = 0; + + // Collect VII cmd and resp queues + for (idx = 0; idx < num_groups; idx++) { + mutex_lock(&groups[idx]->lock); + if (!edgetpu_group_mailbox_detached_locked(groups[idx])) { + sscd_seg_arr[num_queues].addr = + (void *)groups[idx]->vii.cmd_queue_mem.vaddr; + sscd_seg_arr[num_queues].size = groups[idx]->vii.cmd_queue_mem.size; + sscd_seg_arr[num_queues].paddr = + (void *)groups[idx]->vii.cmd_queue_mem.tpu_addr; + sscd_seg_arr[num_queues].vaddr = + (void *)groups[idx]->vii.cmd_queue_mem.vaddr; + num_queues++; + + sscd_seg_arr[num_queues].addr = + (void *)groups[idx]->vii.resp_queue_mem.vaddr; + sscd_seg_arr[num_queues].size = groups[idx]->vii.resp_queue_mem.size; + sscd_seg_arr[num_queues].paddr = + (void *)groups[idx]->vii.resp_queue_mem.tpu_addr; + sscd_seg_arr[num_queues].vaddr = + (void *)groups[idx]->vii.resp_queue_mem.vaddr; + num_queues++; + } + mutex_unlock(&groups[idx]->lock); + } + + // Collect KCI cmd and resp queues + kci = etdev->kci; + sscd_seg_arr[num_queues].addr = (void *)kci->cmd_queue_mem.vaddr; + sscd_seg_arr[num_queues].size = MAX_QUEUE_SIZE * sizeof(struct edgetpu_command_element); + sscd_seg_arr[num_queues].paddr = (void *)kci->cmd_queue_mem.tpu_addr; + sscd_seg_arr[num_queues].vaddr = (void *)kci->cmd_queue_mem.vaddr; + num_queues++; + + sscd_seg_arr[num_queues].addr = (void *)kci->resp_queue_mem.vaddr; + sscd_seg_arr[num_queues].size = MAX_QUEUE_SIZE * + sizeof(struct edgetpu_kci_response_element); + sscd_seg_arr[num_queues].paddr = (void *)kci->resp_queue_mem.tpu_addr; + sscd_seg_arr[num_queues].vaddr = (void *)kci->resp_queue_mem.vaddr; + num_queues++; + + return num_queues; +} diff --git a/drivers/edgetpu/mobile-debug-dump.h b/drivers/edgetpu/mobile-debug-dump.h new file mode 100644 index 0000000..105bd70 --- /dev/null +++ b/drivers/edgetpu/mobile-debug-dump.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Module that defines structure to retrieve debug dump segments + * specific to the family of EdgeTPUs for mobile devices. + * + * Copyright (C) 2021 Google, Inc. + */ +#ifndef __MOBILE_DEBUG_DUMP_H__ +#define __MOBILE_DEBUG_DUMP_H__ + +#include <linux/platform_data/sscoredump.h> + +#include "edgetpu-debug-dump.h" + +struct mobile_sscd_info { + void *pdata; /* SSCD platform data */ + void *dev; /* SSCD platform device */ +}; + +struct mobile_sscd_mappings_dump { + u64 host_address; + u64 device_address; + u64 alloc_iova; + u64 size; +}; + +/* + * Collects the mapping information of all the host mapping and dmabuf mapping buffers of all + * @groups as an array of struct mobile_sscd_mappings_dump and populates the @sscd_seg. + * + * Returns the pointer to the first element of the mappings dump array. The allocated array should + * be freed by the caller after the sscd segment is reported. + * Returns NULL in case of failure. + */ +struct mobile_sscd_mappings_dump * +mobile_sscd_collect_mappings_segment(struct edgetpu_device_group **groups, size_t num_groups, + struct sscd_segment *sscd_seg); + +/* + * Collects the VII cmd and resp queues of all @groups that @etdev belongs to and the KCI cmd and + * resp queues and populates them as @sscd_seg_arr elements. + * + * Returns the total number of queues collected since some queues may have been released for groups + * with detached mailboxes. The return value is less than or equal to the total number of queues + * expected based on @num_groups i.e (2 * @num_groups +2). + */ +size_t mobile_sscd_collect_cmd_resp_queues(struct edgetpu_dev *etdev, + struct edgetpu_device_group **groups, size_t num_groups, + struct sscd_segment *sscd_seg_arr); + +#endif /* MOBILE_DEBUG_DUMP_H_ */ |